How much data is needed for the best machine learning?

March 5, 2024 by 24x7 Offshoring

AI and device getting to know at Capital One

Data. Leveraging standardized cloud systems for data management, model development, and operationalization, we use AI and ML to look out for our clients’ financial properly-being, help them emerge as greater financially empowered, and higher control their spending.

BE equipped for AI built for enterprise data.

There’s quite a few speak about what AI can do. however what can it honestly do on your enterprise? 24x7offshoring business AI gives you all of the AI gear you want and nothing you don’t. And it’s educated in your information so that you realize it’s reliable. revolutionary generation that delivers actual-global consequences. That’s 24x7offshoring business AI.

commercial enterprise AI from 24x7offshoring

relevant

Make agile decisions, free up precious insights, and automate responsibilities with AI designed along with your enterprise context in mind.

dependable

Use AI that is skilled in your enterprise and employer information, driven via 24x7offshoring manner information, and available in the answers you use every day.

responsible

Run accountable AI constructed on leading ethics and information privateness standards even as retaining full governance and lifecycle control throughout your complete organization.

Product advantages

24x7offshoring gives the broadest and deepest set of device learning offerings and helping cloud infrastructure, putting system mastering inside the arms of each developer, statistics scientist and expert practitioner.

textual content-to-Speech
flip textual content into real looking speech.

Speech-to-text
add speech to textual content capabilities to packages.

machine learning
build, teach, and install system learning models speedy.

Translation
Translate textual content the usage of a neural machine translation carrier.

Why 24x7offshoring for AI solutions and services?

groups global are considering how artificial intelligence can help them obtain and enhance commercial enterprise effects. Many executives and IT leaders trust that AI will extensively transform their enterprise inside the subsequent 3 years — however to fulfill the desires of the next day, you ought to put together your infrastructure nowadays. 24x7offshoring main partnerships and expertise will let you enforce AI answers to do simply that.

Generative AI

implementing generative AI solutions calls for careful attention of ethical and privacy implications. but, whilst used responsibly, those technologies have the capacity to significantly beautify productiveness and decrease expenses across a wide variety of packages.

advanced Computing

advanced computing is fundamental to the improvement, education and deployment of AI systems. It gives the computational electricity required to address the complexity and scale of current AI programs and permit advancements in studies, actual-global programs, and the evolution and cost of AI.

Chatbots and large Language models
The talents of chatbots and large language fashions are remodeling the manner corporations function — improving efficiency, enhancing consumer reports and establishing new possibilities throughout diverse sectors.

contact center Modernization
Modernize your touch facilities by introducing automation, improving performance, enhancing consumer interactions and imparting valuable insights for non-stop development. This no longer most effective advantages organizations by way of growing operational efficiency but additionally leads to greater pleasing and personalized virtual experiences for customers.

Predictive Analytics
Predictive analytics supports groups through permitting them to make more accurate choices, reduce dangers, enhance patron stories, optimize operations and gain higher financial results. It has a wide range of packages across industries and is a treasured tool for gaining a aggressive edge in today’s facts-pushed commercial enterprise environment.

information Readiness / Governance
data readiness is vital for the a success deployment of AI in an corporation. It now not simplest improves the overall performance and accuracy of AI models however additionally addresses moral issues, regulatory necessities and operational performance, contributing to the overall success and popularity of AI programs in commercial enterprise settings.

How a good deal records is needed For machine gaining knowledge of?
records is the lifeblood of system mastering. without records, there might be no way to educate and compare 24x7offshoring models. however how an awful lot information do you need for gadget mastering? in this weblog submit, we’re going to discover the factors that have an effect on the amount of information required for an ML assignment, techniques to reduce the quantity of information needed, and guidelines that will help you get started with smaller datasets.

device gaining knowledge of (ML) and predictive analytics are two of the most important disciplines in modern computing. 24x7offshoring is a subset of synthetic intelligence (AI) that focuses on constructing fashions that may study from records rather than relying on specific programming instructions. however, statistics technological know-how is an interdisciplinary field that uses medical strategies, approaches, algorithms, and systems to extract information and insights from structured and unstructured records.

How a great deal facts is needed For machine studying?

picture by using the author: How plenty data is wanted For machine getting to know?
As 24x7offshoring and information science have turn out to be increasingly famous, one of the maximum usually asked questions is: how an awful lot statistics do you want to construct a system mastering version?

the solution to this query depends on numerous elements, together with the

kind of problem being solved,
the complexity of the version,
accuracy of the records,

and availability of categorised facts.
A rule-of-thumb approach indicates that it is best first of all round ten instances extra samples than the variety of capabilities for your dataset.

additionally, statistical strategies together with strength evaluation can help you estimate pattern size for diverse forms of machine-studying problems. aside from accumulating extra information, there are precise strategies to lessen the quantity of statistics wished for an 24x7offshoringversion. these encompass function selection techniques inclusive of 24x7offshoring regression or foremost element analysis (24x7offshoring). Dimensionality discount strategies like autoencoders, manifold learning algorithms, and artificial facts technology strategies like generative adversarial networks (GANs) also are available.

Even though these techniques can assist lessen the amount of information needed for an ML version, it is vital to take into account that exceptional nevertheless matters extra than amount in terms of education a successful model.

How a lot records is wanted?
factors that influence the quantity of records needed
on the subject of developing an powerful gadget learning version, getting access to the proper amount and first-rate of statistics is essential. regrettably, now not all datasets are created identical, and a few might also require extra statistics than others to broaden a successful version. we’ll explore the various factors that have an effect on the quantity of facts wished for gadget learning in addition to strategies to lessen the quantity required.

sort of trouble Being Solved
The kind of problem being solved by means of a machine getting to know model is one of the most important factors influencing the quantity of statistics needed.

as an example, supervised mastering fashions, which require categorised training statistics, will usually need greater statistics than unsupervised models, which do now not use labels.

moreover, positive kinds of troubles, which includes picture reputation or natural language processing (NLP), require large datasets because of their complexity.

The complexity of the version
any other factor influencing the amount of records needed for machine mastering is the complexity of the version itself. The more complex a model is, the greater facts it will require to characteristic successfully and accurately make predictions or classifications. models with many layers or nodes will need extra training records than people with fewer layers or nodes. additionally, fashions that use a couple of algorithms, along with ensemble strategies, will require greater information than people who use handiest a unmarried set of rules.

exceptional and Accuracy of the facts
The first-rate and accuracy of the dataset can also effect how tons statistics is wanted for gadget getting to know. suppose there is lots of noise or wrong information inside the dataset. in that case, it may be vital to increase the dataset size to get correct effects from a device-studying version.

additionally, suppose there are lacking values or outliers in the dataset. in that case, these ought to be either eliminated or imputed for a model to work successfully; thus, growing the dataset length is likewise important.

Estimating the quantity of statistics wanted
Estimating the amount of statistics wished for system studying fashions is important in any statistics technological know-how venture. accurately determining the minimum dataset size required gives records scientists a better knowledge in their ML task’s scope, timeline, and feasibility.

when figuring out the volume of data necessary for an version, elements along with the type of trouble being solved, the complexity of the version, the high-quality and accuracy of the information, and the provision of categorized records all come into play.

Estimating the quantity of information wished may be approached in ways:

A rule-of-thumb approach
or statistical strategies
to estimate sample length.

Rule-of-thumb approach
the rule of thumb-of-thumb technique is maximum usually used with smaller datasets. It includes taking a guess based on beyond reviews and modern expertise. but, it’s miles important to use statistical strategies to estimate sample length with larger datasets. these techniques allow facts scientists to calculate the variety of samples required to make certain sufficient accuracy and reliability in their fashions.

normally speakme, the guideline of thumb regarding machine gaining knowledge of is which you want at the least ten times as many rows (records points) as there are capabilities (columns) to your dataset.

which means if your dataset has 10 columns (i.e., functions), you ought to have as a minimum a hundred rows for premier outcomes.

latest surveys show that around eighty% of a success ML tasks use datasets with greater than 1 million statistics for education functions, with maximum utilising far greater data than this minimum threshold.

ebook a personal demo

book a Demo
information volume & high-quality
whilst determining how lots facts is needed for machine getting to know models or algorithms, you need to consider each the volume and great of the records required.

in addition to assembly the ratio noted above between the number of rows and the quantity of functions, it’s also essential to make certain adequate insurance throughout unique instructions or categories within a given dataset, otherwise called elegance imbalance or sampling bias issues. ensuring a proper amount and fine of suitable education information will help lessen such problems and permit prediction fashions trained in this larger set to gain higher accuracy ratings over the years with out extra tuning/refinement efforts later down the line.

Rule-of-thumb approximately the wide variety of rows in comparison to the wide variety of features helps access-degree information Scientists determine how an awful lot facts they ought to acquire for his or her 24x7offshoring initiatives.

thus ensuring that sufficient input exists whilst implementing system studying techniques can cross a long manner closer to keeping off not unusual pitfalls like pattern bias & underfitting during put up-deployment stages. it’s also assisting reap predictive skills quicker & within shorter improvement cycles, no matter whether one has access to significant volumes of information.

techniques to reduce the amount of records wanted
happily, numerous techniques can lessen the amount of information wished for an 24x7offshoring model. function choice strategies together with essential issue analysis (PCA) and recursive characteristic elimination (RFE) may be used to pick out and cast off redundant features from a dataset.

Dimensionality reduction techniques consisting of singular value decomposition and t-dispensed stochastic neighbor embedding can be used to reduce the quantity of dimensions in a dataset whilst preserving important information.

subsequently, artificial data generation techniques including generative antagonistic networks can be used to generate extra training examples from present datasets.

pointers to lessen the amounts of facts wanted for an 24x7offshoring version
further to using characteristic choice, dimensionality reduction, and artificial statistics era strategies, several different tips can assist entry-degree statistics scientists lessen the quantity of statistics wished for their 24x7offshoring models.

First, they should use pre-educated fashions on every occasion feasible because these models require less education records than custom models built from scratch. second, they should consider the use of transfer studying techniques which permit them to leverage information won from one assignment when fixing another related assignment with fewer education examples.

sooner or later, they have to try special hyperparameter settings considering some settings can also require fewer schooling examples than others.

do not leave out the AI Revolution

From facts to Predictions, Insights and selections in hours.

No-code predictive analytics for regular commercial enterprise users.

try it without spending a dime
Examples of a success tasks with Smaller Datasets
information is an critical issue of any device mastering undertaking, and the quantity of information wished can vary relying at the complexity of the model and the hassle being solved.

but, it is possible to reap a hit outcomes with smaller datasets.

we can now discover a few examples of a success projects finished the usage of smaller datasets. recent surveys have proven that many records scientists can entire a hit initiatives with smaller datasets.

according to a survey conducted by way of Kaggle in 2020, almost 70% of respondents stated they had finished a assignment with fewer than 10,000 samples. additionally, over half of the respondents said that they had finished a project with fewer than five,000 samples.

numerous examples of a hit tasks were completed the usage of smaller datasets. as an example, a team at Stanford college used a dataset of simplest 1,000 pics to create an AI machine that might correctly diagnose pores and skin cancer.

another crew at 24x7offshoring used a dataset of simplest 500 snap shots to create an AI device that might stumble on diabetic retinopathy in eye scans.

those are just examples of the way powerful machine learning fashions can be created using small datasets.

it’s miles certainly feasible to attain successful consequences with smaller datasets for gadget getting to know initiatives.

via utilising function selection techniques and dimensionality reduction strategies, it’s far viable to lessen the quantity of statistics wished for an 24x7offshoring version whilst nevertheless achieving correct outcomes.

See Our solution in movement: Watch our co-founder gift a stay demo of our predictive lead scoring tool in motion. Get a real-time understanding of ways our answer can revolutionize your lead prioritization method.

liberate valuable Insights: Delve deeper into the arena of predictive lead scoring with our comprehensive whitepaper. find out the energy and capability of this sport-changing device in your business. download Whitepaper.

experience it your self: See the electricity of predictive modeling first-hand with a live demo. discover the features, enjoy the user-pleasant interface, and see just how transformative our predictive lead scoring model may be for your enterprise. try stay .

conclusion
on the quit of the day, the amount of records wished for a machine getting to know assignment relies upon on several factors, such as the type of problem being solved, the complexity of the version, the pleasant and accuracy of the facts, and the availability of labeled records. To get an correct estimate of the way a lot records is needed for a given venture, you ought to use either a rule-of-thumb or statistical techniques to calculate pattern sizes. additionally, there are effective techniques to lessen the want for large datasets, consisting of characteristic selection strategies, dimensionality discount techniques, and synthetic records technology strategies.

in the end, a success initiatives with smaller datasets are viable with the right method and to be had technologies.

24x7offshoring observe can help businesses test effects fast in gadget gaining knowledge of. it’s far a powerful platform that utilizes complete information analysis and predictive analytics to help businesses quickly pick out correlations and insights inside datasets. 24x7offshoring notice offers rich visualization tools for evaluating the satisfactory of datasets and models, in addition to clean-to-use computerized modeling capabilities.

With its person-friendly interface, corporations can accelerate the process from exploration to deployment even with restricted technical understanding. This helps them make quicker selections while lowering their costs related to growing system learning packages.

Get Predictive Analytics Powers without a statistics science team

24x7offshoring note robotically transforms your information into predictions and subsequent-high-quality-step techniques, with out coding.

sources:

Device mastering sales Forecast
popular programs of device learning in enterprise
A complete guide to purchaser Lifetime cost Optimization the use of Predictive Analytics
Predictive Analytics in advertising: everything You must know
Revolutionize SaaS revenue Forecasting: release the secrets to Skyrocketing achievement
Empower Your BI groups: No-Code Predictive Analytics for records Analysts
correctly Generate greater Leads with Predictive Analytics and marketing Automation

you can explore all 24x7offshoring models here. This page can be helpful if you are inquisitive about exclusive system learning use instances. sense loose to strive totally free and train your gadget learning version on any dataset with out writing code.

if you ask any data scientist how much facts is wanted for gadget studying, you’ll maximum probably get both “It depends” or “The extra, the higher.” And the aspect is, both solutions are correct.

It honestly depends on the kind of assignment you’re working on, and it’s constantly a brilliant concept to have as many applicable and dependable examples inside the datasets as you could get to get hold of correct outcomes. but the query remains: how an awful lot is sufficient? And if there isn’t sufficient statistics, how will you address its lack?

The revel in with diverse projects that worried synthetic intelligence (AI) and machine studying (ML), allowed us at Postindustria to come up with the most top of the line approaches to technique the statistics quantity difficulty. this is what we’ll communicate approximately inside the study underneath.

The complexity of a version

honestly placed, it’s the quantity of parameters that the algorithm need to learn. The extra capabilities, size, and variability of the expected output it have to keep in mind, the greater records you need to enter. as an instance, you need to train the model to predict housing expenses. you are given a desk where every row is a residence, and columns are the place, the neighborhood, the variety of bedrooms, flooring, bathrooms, etc., and the fee. In this example, you educate the version to predict fees based on the trade of variables in the columns. And to learn how each additional input characteristic affects the input, you’ll want greater facts examples.

The complexity of the mastering set of rules
greater complicated algorithms always require a larger amount of records. in case your undertaking wishes widespread algorithms that use based mastering, a smaller quantity of statistics could be sufficient. Even if you feed the algorithm with greater statistics than it’s enough, the results received’t enhance notably.

The scenario is one of a kind with regards to deep mastering algorithms. unlike conventional system gaining knowledge of, deep gaining knowledge of doesn’t require function engineering (i.e., building enter values for the model to match into) and is still able to examine the illustration from raw information. They work without a predefined shape and determine out all of the parameters themselves. In this case, you’ll want greater records that is relevant for the algorithm-generated classes.

Labeling desires
depending on how many labels the algorithms ought to are expecting, you could need numerous amounts of enter facts. as an example, in case you want to type out the pix of cats from the photographs of the puppies, the algorithm desires to learn some representations internally, and to do so, it converts enter facts into these representations. however if it’s just locating pics of squares and triangles, the representations that the algorithm has to examine are easier, so the amount of statistics it’ll require is much smaller.

suitable errors margin
The type of undertaking you’re operating on is another thing that impacts the quantity of records you need due to the fact one of a kind tasks have extraordinary levels of tolerance for mistakes. as an example, if your venture is to are expecting the weather, the algorithm prediction can be misguided by some 10 or 20%. however when the set of rules ought to inform whether or not the patient has most cancers or no longer, the degree of blunders may cost a little the affected person lifestyles. so you need more data to get more correct outcomes.

input range
In some instances, algorithms need to be taught to characteristic in unpredictable conditions. for instance, when you broaden an online virtual assistant, you evidently need it to recognize what a traveler of a company’s internet site asks. but humans don’t generally write flawlessly correct sentences with standard requests. they may ask hundreds of different questions, use special patterns, make grammar mistakes, and so on. The more out of control the environment is, the greater information you want on your ML undertaking.

based at the elements above, you may outline the scale of information sets you need to acquire properly set of rules overall performance and dependable results. Now allow’s dive deeper and find a solution to our predominant question: how much data is required for gadget gaining knowledge of?

what’s the most beneficial size of AI schooling information sets?
whilst making plans an ML assignment, many fear that they don’t have quite a few statistics, and the outcomes gained’t be as dependable as they can be. however only some sincerely recognise how a lot facts is “too little,” “too much,” or “sufficient.”

The maximum commonplace manner to outline whether a statistics set is sufficient is to apply a 10 instances rule. This rule method that the quantity of enter information (i.e., the wide variety of examples) must be ten instances extra than the quantity of stages of freedom a version has. usually, stages of freedom imply parameters for your statistics set.

So, for example, if your algorithm distinguishes pix of cats from snap shots of dogs based on 1,000 parameters, you need 10,000 pictures to teach the version.

even though the ten times rule in device gaining knowledge of is pretty popular, it can best work for small fashions. large models do no longer observe this rule, as the range of amassed examples doesn’t always reflect the actual amount of schooling statistics. In our case, we’ll want to matter now not most effective the range of rows however the variety of columns, too. The right approach would be to multiply the wide variety of photographs by way of the size of every picture with the aid of the quantity of colour channels.

you could use it for rough estimation to get the assignment off the ground. however to discern out how much facts is needed to educate a specific model inside your particular undertaking, you need to find a technical companion with applicable know-how and visit them.

On top of that, you always have to remember that the AI models don’t observe the records but as a substitute the relationships and patterns in the back of the statistics. So it’s now not only amount in order to have an impact on the results, but also high-quality.

however what are you able to do if the datasets are scarce? There are a few strategies to cope with this trouble.

a way to cope with the dearth of statistics
loss of facts makes it not possible to set up the family members among the input and output records, for that reason inflicting what’s called “‘underfitting”. if you lack input statistics, you could either create synthetic statistics units, increase the existing ones, or observe the information and records generated earlier to a similar hassle. allow’s overview every case in greater element beneath.

records augmentation
facts augmentation is a method of increasing an input dataset by means of slightly converting the prevailing (authentic) examples. It’s extensively used for picture segmentation and category. typical picture alteration strategies consist of cropping, rotation, zooming, flipping, and color modifications.

How a great deal records is required for device studying?
In wellknown, records augmentation facilitates in fixing the hassle of restrained statistics by way of scaling the available datasets. except image classification, it could be utilized in a number of other instances. for instance, right here’s how statistics augmentation works in natural language processing :

back translation: translating the textual content from the authentic language into a goal one after which from target one lower back to authentic
clean data augmentation: changing synonyms, random insertion, random swap, random deletion, shuffle sentence orders to receive new samples and exclude the duplicates
Contextualized phrase embeddings: education the algorithm to use the phrase in distinctive contexts (e.g., while you need to apprehend whether the ‘mouse’ means an animal or a device)

information augmentation adds greater flexible records to the fashions, helps remedy elegance imbalance troubles, and increases generalization potential. but, if the original dataset is biased, so could be the augmented records.

synthetic records generation
synthetic records technology in machine mastering is every so often considered a sort of records augmentation, however these concepts are different. throughout augmentation, we alternate the characteristics of facts (i.e., blur or crop the photograph so we can have 3 images as opposed to one), even as synthetic generation manner creating new facts with alike but no longer similar homes (i.e., growing new snap shots of cats based at the preceding snap shots of cats).

at some stage in artificial information era, you may label the information right away and then generate it from the supply, predicting precisely the records you’ll receive, that’s useful whilst no longer a good deal information is available. but, at the same time as working with the real statistics units, you want to first acquire the facts and then label every instance. This synthetic statistics era technique is widely applied when developing AI-based totally healthcare and fintech answers when you consider that actual-existence data in these industries is challenge to strict privateness legal guidelines.

At Postindustria, we also observe a synthetic information method

Our current virtual jewelry strive-on is a top example of it. To broaden a hand-monitoring model that could work for diverse hand sizes, we’d want to get a pattern of fifty,000-a hundred,000 arms. when you consider that it might be unrealistic to get and label such some of actual snap shots, we created them synthetically by way of drawing the pictures of different arms in numerous positions in a unique visualization program. This gave us the vital datasets for schooling the set of rules to song the hand and make the ring suit the width of the finger.

whilst artificial records can be a great answer for lots projects, it has its flaws.

synthetic statistics vs real facts problem

one of the problems with synthetic information is that it is able to lead to results which have little software in fixing actual-existence problems when real-existence variables are stepping in. for instance, in case you increase a virtual makeup attempt-on using the pics of humans with one pores and skin colour after which generate more synthetic data based on the existing samples, then the app wouldn’t work well on other skin colours. The result? The customers won’t be happy with the characteristic, so the app will reduce the range of capacity shoppers rather than growing it.

some other difficulty of having predominantly synthetic information deals with producing biased effects. the bias may be inherited from the unique sample or when different factors are overlooked. as an instance, if we take ten people with a certain fitness circumstance and create greater information based on the ones instances to expect what number of human beings can increase the identical circumstance out of 1,000, the generated facts might be biased due to the fact the authentic sample is biased by the selection of range (ten).

transfer studying

transfer learning is any other approach of solving the hassle of restrained data. This approach is based totally on applying the knowledge received when operating on one challenge to a new similar venture. The idea of transfer gaining knowledge of is that you teach a neural network on a particular facts set and then use the lower ‘frozen’ layers as characteristic extractors. Then, pinnacle layers are used train different, more specific statistics units.

For example, the version changed into skilled to apprehend photographs of wild animals (e.g., lions, giraffes, bears, elephants, tigers). subsequent, it can extract capabilities from the similarly snap shots to do greater speicifc evaluation and understand animal species (i.e., may be used to distinguish the snap shots of lions and tigers).

How a great deal records is needed for machine learning?

The switch getting to know technique quickens the education degree because it permits you to apply the spine community output as functions in in addition levels. but it can be used simplest while the tasks are comparable; otherwise, this approach can have an effect on the effectiveness of the version.

but, the provision of information itself is frequently not enough to correctly educate an version for a medtech answer. The fine of records is of maximum significance in healthcare initiatives. Heterogeneous information sorts is a assignment to investigate in this discipline. statistics from laboratory assessments, medical photos, vital symptoms, genomics all are available in one of a kind formats, making it hard to installation ML algorithms to all of the information straight away.

another trouble is wide-unfold accessibility of medical datasets. 24x7offshoring, for instance, which is taken into consideration to be one of the pioneers inside the area, claims to have the most effective notably sized database of vital care health data that is publicly available. Its 24x7offshoring database stores and analyzes health information from over forty,000 essential care patients. The information include demographics, laboratory exams, vital symptoms accumulated via patient-worn video display units (blood pressure, oxygen saturation, coronary heart rate), medications, imaging facts and notes written via clinicians. another strong dataset is Truven fitness Analytics database, which records from 230 million patients collected over 40 years based totally on coverage claims. but, it’s not publicly available.

every other problem is small numbers of statistics for some sicknesses. figuring out disorder subtypes with AI calls for a enough amount of facts for each subtype to teach ML fashions. In some instances data are too scarce to train an algorithm. In those cases, scientists try to increase ML fashions that learn as plenty as possible from healthful patient statistics. We must use care, but, to make sure we don’t bias algorithms toward healthy patients.

need facts for an24x7offshoring mission? we are able to get you blanketed!
the size of AI education data sets is crucial for gadget gaining knowledge of initiatives. To outline the most reliable amount of information you need, you have to consider loads of things, inclusive of mission kind, algorithm and model complexity, blunders margin, and enter range. you can also follow a ten instances rule, but it’s now not constantly dependable in relation to complicated responsibilities.

in case you finish that the available facts isn’t sufficient and it’s not possible or too high priced to collect the required actual-world statistics, try to apply one of the scaling techniques. it can be facts augmentation, artificial facts generation, or transfer studying relying on your project desires and finances.

some thing choice you pick out, it’ll want the supervision of experienced facts scientists; otherwise, you risk finishing up with biased relationships among the input and output information. this is where we, at 24x7offshoring, can assist. contact us, and permit’s communicate approximately your 24x7offshoring project!

How to get better data for my AI?

March 4, 2024 by 24x7 Offshoring

How do I get data for my AI? Data. Any engineer who has taken the first steps of gaining knowledge of to paintings with AI techniques has confronted the foremost mission of the space: sourcing sufficient high pleasant facts to make a task feasible. pattern units of information can be had, of direction, however operating … Read more

Unveiling the Essence of Data Labeling: A Comprehensive Guide

June 29, 2024February 24, 2024 by 24x7 Offshoring

In the realm of artificial intelligence and machine learning, data is the cornerstone upon which groundbreaking algorithms and models are built. However, raw data, in its unstructured form, lacks the context and organization necessary for machines to comprehend and derive meaningful insights. This is where data labeling emerges as a crucial process, bridging the gap … Read more

Exploring the World of Data Annotation Services: Types and Applications

February 23, 2024 by 24x7 Offshoring

In today’s data-driven world, the demand for high-quality annotated data is on the rise. Data annotation, the process of labeling data to make it understandable for machines, plays a crucial role in training and improving machine learning models. From image recognition and natural language processing to autonomous vehicles and healthcare, annotated data serves as the … Read more

How to best communicate the results of your data collection to stakeholders?

December 22, 2023 by Lillian

How to best communicate the results of your data collection to stakeholders?

Data Collection

Data collection is a systematic process of gathering observations or measurements. Whether you are performing research for business, governmental or academic purposes, data collection allows you to gain first-hand knowledge and original insights into your research problem.

While methods and aims may differ between fields, the overall process of data collection remains largely the same. Before you begin collecting data, you need to consider:

The aim of the research
The type of data that you will collect
The methods and procedures you will use to collect, store, and process the data

To collect high-quality data that is relevant to your purposes, follow these four steps.

How to effectively communicate the results of the employee engagement report? (with examples)

You did a survey on employee engagement, perfect!

You are already measuring your staff’s commitment to your mission, the team, and their role within the company.

But what are you going to do with the results you get from their contributions?

And most importantly, how will you move from reporting on employee engagement to meeting your staff’s desires for professional growth?

Are you still struggling to find the answers?

Our guide is for you. We have put together practical tips and examples that will allow you to:

Know exactly what to do with employee engagement survey data.
Make sense of what that data reveals.
Excellence in communicating engagement survey results.

The importance of employee engagement for companies

First things first: the relevance and impact of an employee engagement report goes beyond the borders of HR. Of course, measuring engagement is a People Ops function. And it’s also one of the trends in employee engagement: actively listening to determine whether workers are thriving.

But a survey report on employee engagement transcends concerns about team members’ aspirations. It’s a tool full of insightful information that C-suite executives need to understand how healthy and robust their workforce is.

And that is a strategic question. Because there is no way to drive business results without engaged and enthusiastic staff.

Employee engagement drives productivity, performance, and a positive workplace.

How to analyze your employee engagement data

Let’s consider that you already know how to design an employee engagement survey and set your goals.

Therefore, now we will focus on analyzing the results obtained when carrying out the study.

The first important tip is to prepare the analysis in advance. To do this, put in place the mechanisms to quantify and segment the data.

Use numerical scales, scores and percentages

Use numerical scales and convert responses to numerical values whenever you can in your engagement survey.

You will see that comparing the data will not cause you a big headache.

This is because numbers are much less prone to misinterpretation than the opinion of your staff in free text.

Consider qualitative contributions

Although quantitative data is objective, qualitative data must also be analyzed. Because, sometimes, it is not easy to put thoughts and feelings into figures. And without data, it would be impossible to draw conclusions about employee motivation, attitudes, and challenges.

For example, if you ask people to rate their satisfaction with their team from 1 to 5, a number other than five won’t tell you much. But if you ask them to write down the reason for their incomplete satisfaction, you’ll get the gist of their complaint or concern.

Segment the groups of respondents

Without a doubt, your workers are divided into different groups according to different criteria. And that translates into different perceptions of their jobs, their colleagues and their organization.

To find out, segment the engagement survey results while keeping the responses anonymous.

Assure your employees of their anonymity and ask them to indicate their:

age;
gender;
department and team;
tenure;
whether they are junior, intermediate or senior;
your executive level (manager, director, vice president or C-level).

Read the consensus

In this step the real action of the analysis begins.

If you’ve followed our recommendations for quantifying (and segmenting) survey data, you’ll be ready to determine precisely what they’re trying to tell you.

In general terms, if most or all of your staff express the same opinion on an issue, you need to investigate the issue and improve something.

On the other hand, if only a few people are unhappy with the same issue, it may not be necessary to address it so thoroughly.

It all depends on the importance of the topic for the business and the proper functioning of the team and the company. And it’s up to stakeholders to decide whether it’s worth putting effort into finding out what’s behind the results.

Cross data

Engagement levels are not always related to staff’s position, teams or your company. Instead, those levels may have to do with other factors, such as salary or benefits package, to name a couple.

And it’s up to you to combine engagement survey data with data from other sources.

Therefore, consider the information in your HR management system when reviewing the results of your engagement.

Additionally, evaluate engagement data against business information.

This is information that you can extract from your ERP system that is closely related to business results.

Compare results

The ultimate goal of analyzing engagement survey data is to uncover critical areas for improvement. And to do this as comprehensively as possible, you should compare your current engagement survey results with

The results of your previous engagement surveys to understand the engagement levels of your organization, departments and teams over time.
The results of national and global surveys on engagement, especially those of other companies in your sector with a similar activity to yours.

And these are the perceptions to look for:

Why is your organization performing better or worse than before?
Why do certain departments and teams perform better or worse than before?
Why does your company perform better or worse than similar ones in your country or abroad?

How to organize data in your employee engagement report

An employee engagement survey report should shed light on how engagement affects the performance of your company and your staff.

But a report like this is useless if you do not organize the data it contains well. Let’s see how.

First of all, you must keep in mind the objective of the survey. That is developing an action plan to improve the areas with the greatest positive impact on your employee engagement levels.

So keep in mind that traditionally some areas score low in any organization. We’re talking pay and benefits, career progression , and workplace politics, to name a few.

But as a general rule, you should prioritize areas where your company scored poorly compared to industry benchmarks. Those are the ones most likely:

Generate positive ROI once you improve them.
Promote improvement of all other areas of the employee experience .

Typically, the most impactful areas are:

Appreciation to employees;
Response to proactive employees;
Employee participation in decision making;
Communication with leaders.

But it might suggest other areas to focus on.

Now, you need to conveniently organize the survey results in your employee engagement report.

In other words, you must disclose the commitment figures:

for the entire company;
by department;
by team;
by age and sex;
by possession;
by executive level;
by seniority;
by period (current month, quarter or year versus the previous one);
by region (within your country, in your foreign locations and in comparison to national and global benchmarks in your sector);
any combination of the above that makes sense, such as by gender and team or by age and department.

And to identify areas for improvement, you should display the survey data by those areas within each of the divisions above. We recommend that you convert these divisions into distinct sections of the document.

We also recommend using media to visualize results, such as charts and graphs.

For example, use:

Bar charts: to identify trends over time.
Line graphs: to compare this year’s data with last year’s data.
Callout Charts: To highlight surprising figures or conclusions.

These visuals will help stakeholders objectively understand and analyze the results of the employee engagement report. But most importantly, visualization makes it easy to prioritize areas for improvement and provides actionable results.

Good practices for communicating engagement survey results

After collecting and analyzing employee engagement survey data, it’s time to share it within the company.

And here are our tips for communicating engagement survey results to your employees and leaders.

3 Tips for Sharing Employee Engagement Survey Results with Employees

Immediately after completing the employee engagement survey, the CEO should make a communication to the entire company. Alternatively, the VP of HR or a senior HR leader can do this.

And in that communication, they make a recognition.

Thank employees for participating in the study

Your boss – if not yourself – can do this via email or in an all-hands meeting.

But it’s essential that you thank employees as soon as the survey closes. And in addition to saying thank you, the leader must reaffirm his commitment to take engagement to higher levels.

Advise them to appreciate employees’ dedication to helping improve your organizational culture. This will convey the message that employee opinion is valuable, which in itself has a positive impact on engagement.

Briefly present the commitment data you have obtained

One week after closing the survey, your leader should share an overview of the results with the organization. Again, an email or company-wide meeting is all it takes.

The summary should include participation statistics and a summary of the main results (best and worst figures).

This time is also a great opportunity for your leader to explain what employees should expect next. And one way to set expectations is to outline the action plan.

However, their leader cannot provide many details at this time.

The first communication of employee engagement results should focus on numbers with a broader impact.

In other words, it is an occasion to focus on the effect of data at the organizational level.

Report complete engagement data and plan improvements

Three weeks after the survey closes, HR and leaders – team leaders and other executives – must get to work:

Carefully review the results.
Detail the action plan: the areas of improvement you will address and the engagement initiatives you will implement.

Once key stakeholders have decided on the action plan, it’s time to communicate all the details to employees.

The deadline should be no later than one or two months after the survey closes.

3 Tips for Sharing Engagement Survey Results with Leaders and Key Stakeholders

Once the results of the engagement survey are obtained, the first step is to share them with the management team. These are our main recommendations on how to approach this task.

There is no need to rush when deciding what to do with the data.

Give your leaders time to review the engagement data, digest it, and think carefully about it.

We recommend this calendar:

One week after the survey closes, before communicating high-level results.
Three weeks after completing the survey, before we begin to thoroughly discuss the data and delve into the action plan.
One or two months after the survey closes, before communicating detailed results.

Increasing the engagement levels of your employees is a process of change. And as with any corporate change, internalizing it does not happen overnight. Additionally, your leaders are the ones who must steer the helm of change, so they need time to prepare.

Emphasize the end goal and fuel the dialogue

The process of scrutinizing engagement data starts with you. Introduce your leaders:

the overall employee engagement score;
company-wide trends;
department-specific trends;
strengths and weaknesses (or opportunities).

Leaders must clearly understand what organizational culture is being pursued.

And the survey results will help them figure out what’s missing.

So make sure you communicate this mindset to them.

Next, as you discuss the data in depth, you should promote an open dialogue. Only then will your leaders agree on an effective action plan to increase engagement levels.

Don’t sweep problems under the rug

You can’t increase employee engagement without transparency. And you play a role in it too.

You should share the fantastic results and painful numbers from your engagement survey with your leaders.

Reporting alarming discoveries is mandatory for improvement.

After all, how could you improve something without fully understanding and dissecting it?

Additionally, investigating negative ratings and comments is ultimately a win-win for both workers and the company.

Real Examples of Employee Engagement Reports

Here are four employee engagement reports that caught our attention. We’ll investigate why they did it and what you can learn from them.

1. New Mexico Department of Environment

The New Mexico Department of Environment’s engagement report :

1. Start with a message from a senior leader in your organization, providing:

The overall response rate;
The overall level of commitment;
Some areas for improvement;
A reaffirmation of senior management’s commitment to addressing employee feedback.

2. Use graphs to highlight the most interesting conclusions.

3. Breaks down the figures from an overview to year-on-year highs and lows by department and survey section .

4. Compare your employees’ level of engagement to a national benchmark.

5. Includes information on the organization’s commitment actions throughout the year prior to the survey.

6. Demographic breakdown of respondents .

7. Clarify next steps for your leaders and how employees can participate.

8. Discloses an appendix containing year-over-year scores for all survey questions that used a numerical scale.

Note: The year-over-year comparison allows this organization to identify trends in employee engagement.

2. GitLab

The GitLab Commit Report :

Explain how survey responses will be kept confidential .
List the areas of interest for the survey.
Shows a chronology of the actions that the company carried out around the survey.
Clarify the steps that will follow after closing the survey.
It presents the global response rate, the global commitment level and an industry benchmark.
Thank you employees for participating in the survey.
It reveals the top-ranked responses in the three main areas of interest and compares them to the industry benchmark.
Highlight areas that require improvement.

👀 Note: The calendar with the survey actions may seem insignificant. However, it is an element of transparency that generates trust among readers of the report.

3. UCI Irvine Human Resources

The University of California HR Employee Engagement Report :

Start by justifying why employee engagement is important to workplace culture and various stakeholders.
Defines the responsibilities of everyone involved in engaging those stakeholders.
Remember the results of the previous employee engagement survey and set them as a reference .
Compare the most recent survey results with the baseline figures.
Distinguishes engaged, disengaged, and actively disengaged staff members between previous and most recent data by organizational unit.
Lists new opportunities that the department should address and strengths that it should continue to explore.
It presents a timeline of the phased engagement program and some planned actions.
Describes the next steps leaders should take with their team members.

👀 Note: The report notes that the figures vary between the two editions of the survey because the HR department encouraged staff participation instead of forcing it.

4. UCI Riverside Chancellor’s Office

The University of California, Riverside Office of the Chancellor’s Employee Engagement Report :

Contains instructions on how scores were calculated .
Compare employee engagement survey results to different types of references , from previous survey results to national numbers.
Highlights issues that represent a priority for the organization.
Distinguish the level of statistical significance that each number has , clarifying the extent to which they are significant.
Describe the suggested actions in some detail.
It groups scores by category – such as professional development or performance management -, role – such as manager or director -, gender, ethnicity, seniority and salary range.
Break down the scores within each category.
Shows the total percentage of employees at each engagement level , from highly engaged, empowered and energized to disengaged.
It concludes with the main drivers of commitment , such as the promotion of social well-being.

Note: The document is very visual and relies on colors to present data. While this appeals to most readers, it makes it less inclusive and compromises organization-wide interpretation.

Now that you know how to analyze your survey data and organize your engagement report, learn how to create an employee engagement program .

How to better manage data validation and cleaning processes?

December 20, 2023 by Lillian

How to better manage data validation and cleaning processes? Data Data is a collection of facts, figures, objects, symbols, and events gathered from different sources. Organizations collect data with various data collection methods to make better decisions. Without data, it would be difficult for organizations to make appropriate decisions, so data is collected from … Read more

How to best handle data storage and archiving after the project is finished?

December 13, 2023 by Lillian

How to best handle data storage and archiving after the project is finished?

What is Data Collection?

Data collection is the procedure of collecting, measuring, and analyzing accurate insights for research using standard validated techniques.

To collect data, we must first identify what information we need and how we will collect it. We can also evaluate a hypothesis based on collected data. In most cases, data collection is the primary and most important step for research. The approach to data collection is different for different fields of study, depending on the required information.

Research Data Management ( RDM) is present in all phases of research and encompasses the collection, documentation, storage and preservation of data used or generated during a research project. Data management helps researchers: organize it, locate it, preserve it, reuse it.

Additionally, data management allows:

Save time and make efficient use of available resources : You will be able to find, understand and use data whenever you need.
Facilitate the reuse of the data you have generated or collected: Correct management and documentation of data throughout its life cycle will allow it to remain accurate, complete, authentic and reliable. These attributes will allow them to be understood and used by other people.
Comply with the requirements of funding agencies : More and more agencies require the presentation of data management plans and/or the deposit of data in repositories as requirements for research funding.
Protect and preserve data : By managing and depositing data in appropriate repositories, you can safely safeguard it over time, protecting your investment of time and resources and allowing it to serve new research and discoveries in the future.

Research data is “all that material that serves to certify the results of the research that is carried out, that has been recorded during it and that has been recognized by the scientific community” (Torres-Salinas; Robinson-García; Cabezas-Clavijo, 2012), that is, it is any information collected, used or generated in experimentation, observation, measurement, simulation, calculation, analysis, interpretation, study or any other inquiry process that supports and justifies the scientific contributions that are disseminated in research publications.

They come in any format and support, for example:

Numerical files, spreadsheets, tables, etc.
Text documents in different versions
Images, graphics, audio files, video, etc.
Software code or records, databases, etc.
Geospatial data , georeferenced information

Joint Statement on Research Data from STM, DataCite and Crossref

In 2012, DataCite and STM drafted an initial joint statement on linking and citing research data.

The signatories of this statement recommend the following as best practices in research data sharing:

When publishing their results, researchers deposit the related research data and results in a trusted data repository that assigns persistent identifiers (DOIs when available). Researchers link to research data using persistent identifiers.
When using research data created by others, researchers provide attribution by citing the data sets in the references section using persistent identifiers.
Data repositories facilitate the sharing of research results in a FAIR manner, including support for metadata quality and completeness.
Editors establish appropriate data policies for journals, outlining how data will be shared along with the published article.
The editors establish instructions for authors to include Data Citations with persistent identifiers in the references section of articles.
Publishers include Data Citations and links to data in Data Availability Statements with persistent identifiers (DOIs when available) in the article metadata recorded in Crossref.
In addition to Data Citations, Data Availability Statements (human and machine readable) are included in published articles where applicable.
Repositories and publishers connect articles and data sets through persistent identifier connections in metadata and reference lists.
Funders and research organizations provide researchers with guidance on open science practices, track compliance with open science policies where possible, and promote and incentivize researchers to openly share, cite, and link research data.
Funders, policy-making institutions, publishers, and research organizations collaborate to align FAIR research data policies and guidelines.
All stakeholders collaborate to develop tools, processes and incentives throughout the research cycle to facilitate the sharing of high-quality research data, making all steps in the process clear, easy and efficient for researchers through provision of support and guidance.
Stakeholders responsible for research evaluation factor data sharing and data citation into their reward and recognition system structures.

The first phase of an investigation requires designing and planning your project. To do this, you must:

Know the requirements and programs of the financing agencies
Search research data
Prepare a Data Management Plan .

Other prior considerations:

If your research involves working with humans, informed consent must be obtained.
If you are involved in a collaborative research project with other academic institutions, industry partners or citizen science partners, you will need to ensure that your partners agree to the data sharing.
Think about whether you are going to work with confidential personal or commercial data.
Think about what systems or tools you will use to make data accessible and what people will need access to it.

During the project…

This is the phase of the project where the researcher organizes, documents, processes and stores the data.

Is required :

Update the Data Management Plan
Organize and document data
Process the data
Store data for security and preservation

The description of data must provide a context for its interpretation and use, since the data itself lacks this information, unlike scientific publications. It is about being able to understand and reuse them .

The following information should be included:

The context: history of the project, objectives and hypotheses.
Origin of the data: if the data is generated within the project or if it is collected (in this case, indicate the source from which it was extracted).
Collection methods, instruments used.
Typology and format of data (observational, experimental, computational data, etc.)
Description standards: what metadata standard to use.
Structure of data files and relationships between files.
Data validation, verification, cleaning and procedures carried out to ensure its quality.
Changes made to the data over time since its original creation and identification of the different versions.
Information about access, conditions of use or confidentiality.
Names, labels and description of variables and values.

Template for a data dictionary
READme file template for a data set
Cornell University
REadme file template (Madroño consortium)

project

STRUCTURE OF A DATASET

The data must be clean and correctly structured and ordered:

A data set is structured if:

Each variable forms a column
Each observation forms a row
Each cell is a simple measurement

Some recommendations :

Structure the data in TIDY (vertical) format i.e. each value is a row, rather than horizontally. Non-TIDY (horizontal) data.
Columns are used for variables and their names can be up to 8 characters long without spaces or special signs.
Avoid text values to encode variables, better encode them with numbers .
In each cell, a single value
If you do not have a value available , provide the missing value codes.
Provide data tables , which collect all the data encodings and denominations used.
Use data dictionary or separate list of these short variable names and their full meaning

DATA SORTING

Ordered data or “TIDY DATA” are those obtained from a process called “DATA TIDYING” or data ordering. It is one of the important cleaning processes during big data processing.

Ordered data sets have a structure that makes work easier; They are easy to manipulate, model and visualize. ‘Tidy’ data sets are arranged in such a way that each variable is a column and each observation (or case) is a row.” (Wikipedia).

Practical Guide to Publishing Tabular Data to CSV Files
Content prepared by Carlos de la Fuente García, expert in open data, within the framework of the Aporta Initiative, of the Ministry of Economic Affairs and Digital Transformation
Data Quality Review Checklist
For research data during the life of the project it is recommended:
- Store data in readable formats for the long term.
- Check the files from time to time.
- Clearly organize and label stored data so that it is easily findable and accessible.
- Take into consideration the physical degradation of optical and magnetic media in case it is necessary to copy or migrate data.
- Store data on different media, even for a short-term project, for example on the hard drive and on CD.
- Create digital versions of paper documentation, in PDF/A Format for long-term preservation and storage.
- Take into account support conservation factors, such as changes in temperature, relative humidity, light, etc.
Storage media are complementary to each other and some of the most common options are:
- Personal or project data storage (using USB drives, computer hard drives or network drives within the institution), recommended only for use in the course of research.
- Institutional repository (Dehesa or that of your university).
- National data archiving services.
- Cloud data warehouse.
- Repositories (RIO, Zenodo, disciplinary repositories).
The repository selected to store the data must guarantee its long-term preservation, and ease of recovery and access.
When choosing a repository, some aspects must be taken into account:
According to the OpenAIRE Research Data Management Briefing Paper, data should be deposited in a data repository according to the following order of preference :
1. Multidisciplinary data repository. Zenodo, Dryad, Dataverse, Figshare, Mendeley Data,
2. Other data repositories
3. Together with the scientific publications ODYSSEY
REPOSITORY FINDERS
Repository search engines could also help you:
- Re3data: global registry of research data repositories from different academic disciplines. Managed and maintained by DataCite .
- Fairsharing : search engine for standards, data repositories and open access policies in all disciplines.
- Repository Finder – Allows researchers to search for the most suitable repository to deposit data.
Consolidated thematic data repository for that discipline. Re3data or Data Repositories, for example, in Social Sciences, World Values Survey or Wellcome Library in History of Medicine Institutional data repository. Digital CSIC or Harvard Dataverse, Dehesa (repository of the University of Extremadura).

There may be exceptions to open dissemination, based on reasons of confidentiality, privacy, security, industrial exploitation, etc. (H2020, Work Programme, Annexes, L Conditions related to open access to research data).

There are some reasons why certain types of data cannot and/or should not be shared , either in whole or in part, for example:

When the data constitutes or contains sensitive information . There may be national and even institutional regulations on data protection that will need to be taken into account. In these cases, precautions must be taken to anonymize the data and, in this way, make its access and reuse possible without any errors in the ethical use of the information.
When the data is not the property of those who collected it or when it is shared by more than one party, be they people or institutions . In these cases, you must have the necessary permissions from the owners to share and/or reuse the data.
When the data has a financial value associated with its intellectual property , which makes it unwise to share the data early. Before sharing them, you must verify whether these types of limits exist and, according to each case, determine the time that must pass before these restrictions cease to apply.

How to best verify the accuracy of self-reported data?

December 11, 2023 by Lillian

How to best verify the accuracy of self-reported data?

Data

What is Data Collection?

Data collection is the procedure of collecting, measuring, and analyzing accurate insights for research using standard validated techniques.

Put simply, data collection is the process of gathering information for a specific purpose. It can be used to answer research questions, make informed business decisions, or improve products and services.

A significant number of scientific investigations denote a lack of rigor, and this is largely due to the non-validation of the instruments used. This is much more evident in the behavioral sciences, where the most frequent methodology is qualitative, a type of research where an indiscriminate use of instruments is observed, which are not typical of this methodology. This responds to an interest in the search for contextualization and homogeneity.

In an analysis carried out on 102 doctoral theses developed in the last 10 years, it was detected that the most used instrument is the survey; that each investigation designed its own instrument; and that, in the best of cases, responded to the objectives set (Conference in postdoctoral course: Analysis of the use of instruments in doctoral research, presented in 2014 by Tomás Crespo Borges, at the Pedagogical University of Villa Clara).

Due to its importance and complexity of application, instrument validation is considered a type of study within intervention studies, that is, at the same level as experimental, quasi-experimental, among others.

The questionnaire is an instrument for collecting information, designed to quantify and universalize it. For this reason, the moment of validation is of great importance, since the results obtained from its application can falsify the research, and thus, lead to fatal consequences in robust studies, in the social, constructive, and life of a patient. , among others.

In this work, dividing sections will be used; in practice it is a process that is presented as a system, where all its elements have an important function.

A first conception that has two phases is described below:

Phase 1: Generalities of validation

An instrument must meet two fundamental elements: validity and reliability, to match the gold standard instrument. If it does not exist, then it must meet a series of requirements to be reliable enough to accept the results in scientific research.

First: Validation involves two fundamental concepts, what has been applied up to this point? Is it good, surely? Second: How accurate is the new instrument to compare it with the one accepted by the scientific community, as correct in its measurements?

Phase 2: Internal validity

Validity is the degree to which an instrument measures what it is intended to measure. To obtain it, the instrument to be used must be compared with the ideal, gold standard or Gold Standard .

Reaffirmed as a process, five sources of evidence have been postulated for it: according to the content, the internal structure, in relation to other variables, in the consequences of the instrument and in the response processes.

Reliability is the degree of congruence with which an instrument measures the variable. It is obtained by evaluating reproducibility, which is when there is a good correlation in the measurements at different times; and on the other hand, reliability, which is the accuracy of the measurements at different times. The application of both concepts is revealed in a recent article, where an instrument is validated with the purpose of being used in a study on tourist destinations in the province of El Oro, Ecuador.

When exploring the state of the art, the first thing to do is verify the existence of instruments applied in previous research, used for the same purpose, that have been validated at the time, as part of the investigative process. The most used tests, depending on the measurements of the variables, can be Student’s t or Anova, if the data follow a normal distribution; otherwise, their non-parametric counterparts; Wilcoxon or Kruskal Wallis, in the case of two or three measurements, respectively, in both situations.

When there is no instrument that fits the objectives of the research, then it must be formed and contrasted with the ideal or gold standard.

In the second option, validity is very difficult to prove, since it has been decided to use an instrument different from those existing in the literature consulted.

Next, reliability is verified. For this, reproducibility is measured . The instrument is applied several times (two or more) in samples that belong to the same universe or population where the research is carried out. To obtain a correlation considered good in the results (according to the Pearson, Spearman coefficients or the CCC coefficient of agreement) between the measurements, a value greater than 0.7 is accepted, although the ideal is 0.9.

For reliability, it is proven that in the different measurements, taken in the same universe or population, the responses of the subjects do not differ significantly, that is, there is accuracy in the instrument measurements at different times. The most used statistical tests are Aiken’s V and Dahlberg’s error. Therefore, validity is measured with another instrument, and reliability with the same one.

Other authors include the term optimization. It is associated with minimizing the error when providing a criterion, at the time of decision-making, based on the results obtained from the instrument.

In general sense, in the studies discussed it can be seen that there are several ways to carry out the validation of measurement instruments. The one that the researcher considers most appropriate can be used, but keeping in mind that the one selected meets all the necessary scientific rigor.

Below, a methodology will be shown to validate a measurement instrument, which is a hybrid between the conception of two different groups of authors, who are essentially similar.

Qualitative, which coincides with content analysis, is part of internal validity. To this are added the reliability and the construct, which belong to the quantitative, as well as the criterion, stability and performance. These last three correspond to external validity.

A second conception, which has six phases, in correspondence with Supo ‘s idea , is described below:

Phase 1 : qualitative or content validation. It is part of internal validity. It is the creation of the instrument. It is divided into three moments, which do not have to follow an order, but are mandatory. It coincides with a type of diagnostic investigation.

Approach to the population: its purpose is to investigate the problem being addressed, approach the units of analysis or variables that should be used in the research. To do this, interviews, population survey studies and others can be carried out to provide this information.
Expert judgment: the selected experts are responsible for assessing whether the items in the instrument are clear, precise, relevant, coherent and exhaustive.
Rational validity (knowledge): they must be concepts that have been searched in the literature. It is assumed that the researcher is knowledgeable about the topic being studied.

Phase 2 : quantitative or reliability. It is within the internal validity of the instrument.

This phase was detailed previously. According to Aiken : “…strictly speaking, rather than being a characteristic of a test, reliability is a property of the scores obtained when the test is administered to a particular group of people, on a particular occasion, and under specific conditions.”

How to best handle outliers or anomalous data points?

December 6, 2023 by Lillian

How to best handle outliers or anomalous data points?

DATA

Data is a collection of facts, figures, objects, symbols, and events gathered from different sources. Organizations collect data with various data collection methods to make better decisions. Without data, it would be difficult for organizations to make appropriate decisions, so data is collected from different audiences at various points in time.

Outlier detection in the field of data mining (DM) and knowledge discovery from data (KDD) is of great interest in areas that require decision support systems, such as, for example, in the financial area, where through DM you can detect financial fraud or find errors produced by users. Therefore, it is essential to evaluate the veracity of the information, through methods for detecting unusual behavior in the data.

This article proposes a method to detect values that are considered outliers in a database of nominal type data. The method implements a global “k” nearest neighbors algorithm, a clustering algorithm called k-means , and a statistical method called chi-square. The application of these techniques has been implemented on a database of clients who have requested financial credit. The experiment was performed on a data set with 1180 tuples, where outliers were deliberately introduced. The results demonstrated that the proposed method is capable of detecting all introduced outliers.

Detecting outliers represents a challenge in data mining techniques. Outliers or also called anomalous values have different properties with respect to generality, since due to the nature of their values and therefore, their behavior, they are not data that maintain a behavior similar to the majority. Anomalous are susceptible to being introduced by malicious mechanisms ( Atkinson, 1981 ).

Mandhare and Idate (2017) consider this type of data to be a threat and define it as irrelevant or malicious. Additionally, this data creates conflicts during the analysis process, resulting in unreliable and inconsistent information. However, although anomalous data are irrelevant for finding patterns in everyday data, they are useful as an object of study in cases where, through them, it is possible to identify problems such as financial fraud through an uncontrolled process.

What is anomaly detection?

Anomaly detection examines specific data points and detects unusual occurrences that appear suspicious because they are different from established patterns of behavior. Anomaly detection is not new, but as the volume of data increases, manual tracking is no longer practical.

Why is anomaly detection important?

Anomaly detection is especially important in industries such as finance, retail, and cybersecurity, but all businesses should consider implementing an anomaly detection solution. Such a solution provides an automated means to detect harmful outliers and protects data. For example, banking is a sector that benefits from anomaly detection. Thanks to it, banks can identify fraudulent activity and inconsistent patterns, and protect data.

Data is the lifeline of your business and compromising it can put your operation at risk. Without anomaly detection, you could lose revenue and brand value, which takes years to cultivate. Your company is facing security breaches and the loss of confidential customer information. If this happens, you risk losing a level of customer trust that may be irretrievable.

The detection process using data mining techniques facilitates the search for anomalous values ( Arce, Lima, Orellana, Ortega and Sellers, 2018 ). Several studies show that most of this type of data also originates from domains such as credit cards ( Bansal, Gaur, & Singh, 2016 ), security systems ( Khan, Pradhan, & Fatima, 2017 ), and electronic health information ( Zhang & Wang, 2018 ).

The detection process includes a data mining process that uses tools based on unsupervised algorithms (Onan, 2017). The detection process consists of two approaches depending on its form: local and global ( Monamo, Marivate, & Twala, 2017 ). Global approaches include a set of techniques in which each anomaly is assigned a score relative to the global data set. On the other hand, local approaches represent the anomalies in a given data with respect to its direct neighborhood; that is, to the data that are close in terms of the similarity of their characteristics.

According to the aforementioned concepts, the local approach detects outliers that are ignored when a global approach is used, especially those with variable density (Amer and Goldstein, 2012). Examples of such algorithms are those based on i) clustering and ii) nearest neighbor. The first category algorithm considers outliers to be in sparse neighborhoods, which are far from the nearest neighbors. While the second category operates in grouped algorithms ( Onan, 2017 ).

There are several approaches related to the detection of outliers, in this context, Hassanat, Abbadi, Altarawneh, and Alhasanat (2015) , carried out a survey where a summary of the different studies of detection of outliers is presented, these being: statistics-based approach, the distance-based approach and the density-based approach. The authors present a discussion related to outliers, and conclude that the k-mean algorithm is the most popular in clustering a data set.

Furthermore, in other studies (Dang et al., 2015; Ganji, 2012; Gu et al., 2017; Malini and Pushpa, 2017; Mandhare and Idate, 2017; Sumaiya Thaseen and Aswani Kumar, 2017; Yan et al., 2016 ) data mining techniques, statistical methods or both are used. For outlier detection, nearest neighbor (KNN) techniques have commonly been applied along with others to find unusual patterns during data behavior or to improve process performance. What’s up. (2017) present an efficient grid-based method for finding outlier data patterns in large data sets.

Similarly, Yan et al. (2016) propose an outlier detection method with KNN and data pruning, which takes successive samples of tuples and columns, and applies a KNN algorithm to reduce dimensionality without losing relevant information.

Classification of significant columns

To classify the significant columns, the chi-square statistic was used. Chi-square is a non-parametric test used to determine whether a distribution of observed frequencies differs from expected theoretical frequencies ( Gol and Abur, 2015 ). The weight of the input column (columns that determine the customer profile) is calculated in relation to the output column (credit amount). The higher the weight of a column corresponding to the input columns on a scale of zero to one, the more relevant it is considered.

That is, the closer the weight value is to one, the more important the relationship with respect to the output column will be. The statistic can only be applied to nominal type columns and has been selected as a method to define relevance. Chi-square reports a level of significance of the associations or dependencies and was used as a hypothesis test on the weight or importance of each of the columns with respect to the output column S. The resulting value is stored in a column called weight , which together with the anomaly score is reported at the end of the process.

Nearest local neighbor rating

To obtain the values with suspected abnormality, K-NN Global Anomaly Score is used. KNN is based on the k-nearest neighbor algorithm, which calculates the anomaly score of the data relative to the neighborhood. Usually, outliers are far from their neighbors or their neighborhood is sparse. In the first case, it is known as global anomaly detection and is identified with KNN; The second refers to an approach based on local density.

The score comes by default from the average of the distance to the nearest neighbors ( Amer and Goldstein, 2012 ). In the k nearest neighbor classification, the output column S of the nearest neighbor of the training dataset is related to a new data not classified in the prediction, this implies a linear decision line.

To obtain a correct prediction, the value k (number of neighbors to be considered around the analyzed value) must be carefully configured. A high value of k represents a poor solution with respect to prediction, while low values tend to generate noise ( Bhattacharyya, Jha, Tharakunnel, & Westland, 2011 ).

Frequently, the parameter k is chosen empirically and depends on each problem. Hassanat, Abbadi, Altarawneh, and Alhasanat (2014) propose carrying out tests with different numbers of close neighbors until reaching the one with the best precision. Their proposal starts with values from k=1 to k= square root of the number of tuples in the training dataset. The general rule is often to map k with the square root of the number of tuples in dataset D.

HOW CAN WE SOLVE THE PROBLEM OF ATYPICAL DATA?

If we have confirmed that these outliers are not due to an error when constructing the database or in measuring the variable, eliminating them is not the solution. If it is not due to an error, eliminating or replacing it can modify the inferences made from that information, because it introduces a bias, reduces the sample size, and can affect both the distribution and the variances.

Furthermore, the treasure of our research lies in the variability of the data!

That is, variability (differences in the behavior of a phenomenon) must be explained, not eliminated. And if you still can’t explain it, you should at least be able to reduce the influence of these outliers on your data.

The best option is to remove weight from these atypical observations using robust techniques .

Robust statistical methods are modern techniques that address these problems. They are similar to the classic ones but are less affected by the presence of outliers or small variations with respect to the models’ hypotheses.

ALTERNATIVES TO THE MEDIA

If we calculate the median (the center value of an ordered sample) for the second data set we have a value of 14 (the same as for the first data set). We see that this centrality statistic has not been disturbed by the presence of an extreme value, therefore, it is more robust.

Let’s look at other alternatives…

The trimmed mean (trimming) “discards” extreme values. That is, it eliminates a fraction of the extreme data from the analysis (eg 20%) and calculates the mean of the new data set. The trimmed mean for our case would be worth 13.67.

The winsorized mean progressively replaces a percentage of the extreme values (eg 20%) with less extreme ones. In our case, the winsorized mean of the second sample would be the same 13.62.

We see that all of these robust estimates better represent the sample and are less affected by extreme data.

What measures are in place to ensure the security of your data?

December 4, 2023 by Lillian

What measures are in place to ensure the security of your data?

Data

For instance, an organization must collect data on product demand, customer preferences, and competitors before launching a new product. If data is not collected beforehand, the organization’s newly launched product may fail for many reasons, such as less demand and inability to meet customer needs.

Although data is a valuable asset for every organization, it does not serve any purpose until analyzed or processed to get the desired results.

A phrase attributed to the creator of the Internet, Tim Berners-Lee, is that “data is precious and will last longer than the systems themselves.”

The computer scientist was referring to the fact that information is something highly coveted, one of the most valuable assets that companies have, so it must be protected. The loss of sensitive data can represent the bankruptcy of a company.

Faced with an increasing number of threats, it is necessary to implement measures to protect information in companies. And before doing so, it is necessary to classify the data you have and the risks to which you are exposed: the price list of the products that are marketed is not the same as the estimated sales figure that is planned to be achieved in the year or base. of customer data.

To talk about the cloud these days is to talk about a need for storage, flexibility, connectivity and decision making in real time. Information is a constantly growing asset and needs to be managed by work teams, and platforms such as Claro Drive Negocio offer, in addition to that storage space, collaboration tools to manage an organization’s data.

In cloud storage, the user, instead of saving the data on their computers or hard drives, does so somewhere in the remote location, which can be accessed through the internet service. There are several providers of these services that sell space on the network for different ranges, but few offer true security and protection of that gold that you have in your company called: data.

To give you context, more than a third of companies have consolidated flexible and scalable cloud models as an alternative to execute their workload and achieve their digital transformation, reducing costs. Hosted information management services allow IT to maintain control and administrators to monitor access and hierarchies by business units.

Five key security measures

Below are five security recommendations to protect information in companies:

Make backup copies or backups . Replicating or having a copy of the information outside the company’s facilities can save your operation in the event of an attack. In this case, options can be sought in the cloud or in data centers so that the protected information is available at any time. It is also important that the frequency with which the backup is made can be configured, so that the most recent data is backed up.
Foster a culture of strong passwords . Kaspersky recommends that passwords be longer than eight characters, including uppercase, lowercase, numbers, and special characters. The manufacturer also suggests not including personal information or common words; use a password for each service ; change them periodically; Do not share them, write them on paper or store them in the web browser. Every year, Nordpass publishes a ranking of the 200 worst passwords used in the world. The worst four are “123456”, “123456789”, “picture1” and “password”.
Protect email. Now that most of the communication is done through this medium, it is advisable to have anti-spam filters and message encryption systems to protect and take care of the privacy of the data. Spam filters help control the receipt of unsolicited emails, which may be infected with viruses and potentially compromise the security of company data.
Use antivirus. This tool should provide protection against security threats such as zero-day attacks, ransomware, and cryptojacking. And it must also be installed on cell phones that contain company information.
Control access to information. One way to minimize the risk and consequent impact of errors on data security is to provide access to data according to the profile of each user. With the principle of least privilege, it is considered that, if a person does not have access to certain vital company information, he cannot put it at risk.

security measures

In security, nothing is too much

In a synthetic way, the National Cybersecurity Institute of Spain, INCIBE, recommends the following “basic security measures”:

Keep systems updated free of viruses and vulnerabilities
Raise awareness among employees about the correct use of corporate systems
Use secure networks to communicate with customers, encrypting information when necessary
Include customer information in annual risk analyses, perform regular backups, and verify your restore procedures
Implement correct authentication mechanisms, communicate passwords to clients securely and store them encrypted, ensuring that only they can recover and change them

The first time a company or business faces the decision to automate a process it can be somewhat intimidating, however, taking into account the following points, it is a simple task.

1.- Start with the easy processes

Many companies start considering automation because they have a large, inflexible process that they know takes up too much time and money. So they start with their most complex problem and work backwards. This strategy is generally expensive and time-consuming, what you should do is review your most basic processes and automate them first. For example, are you emailing a document with revisions when you should be building an automated workflow? There are probably dozens, if not hundreds of these simple processes that you can address and automate before taking on your “giant” process.

2.- Make sure your employees lose their fear of automation

Many times an employee who is not familiar with an automated process is afraid of it. Because? In general he is afraid that automation will eliminate his position. That’s why it’s important to build a supportive culture around automation and get your employees to understand that just because some of their work is now being assisted by an automated process , it doesn’t mean they are any less valuable.