How would you feel if a search engine returns irrelevant results for your search query? I am sure you won’t like it. What if I tell you that you can make money by helping a search engine in decision making process like which result is relevant and which is not? In this article, we will talk about UHRS, a crowsourcing platform where you can complete simple small tasks and earn a lot of money. For an example, you will be given 2 links along with a search query. You just have to judge which link is more suitable to the query and you get paid for your judgement. UHRS can be a great way to make some extra money working from home.
I will try to answer following questions about UHRS in this article –
- What is UHRS?
- How to get access to UHRS?
- What kind of work UHRS offers?
- How to Work on UHRS and make money?
- How much money one can earn on UHRS?
- How to withdraw earnings from UHRS?
- How to get most out of UHRS?
- UHRS terminology
What is UHRS?
UHRS, Universal Human Relevance System is a crowdsourcing platform by Microsoft used for data labeling with the help of millions of people around the world it calls ‘Judges’. Judges/workers can work from home and get paid for doing simple tasks such as text annotation, image annotation, data collection, search relevance tasks and much more. Basically UHRS provides human intelligence to train AI applications/tools, products and machine learning models.
How to Sign Up/Access UHRS?
UHRS has partnered with different vendors/websites to hire workers/judges. These companies act as a door between you and UHRS. You can only access UHRS through one of such door. All the money you make in a week completing tasks on UHRS is transferred to website you used to enter UHRS through. This way these websites handles billing/invoices and payment for UHRS.
List of websites to access UHRS and how to join UHRS through them –
Clickworker – Registered users on Clickworker can access UHRS jobs. To get access to UHRS jobs on Clickworker, follow these steps –
- Join Clickworker and log into your account.
- On dashboard you will see a banner about UHRS saying Qualify for UHRS. Or you can check a job listing about UHRS under available jobs tab. A link will lead you to the UHRS But before you can get your UHRS account, you must complete assessments UHRS I and UHRS II available in assessment section in your Clickworker account.
- You will also need to create a Live ID. Clickworker guides you through the whole process. Remember the access remains valid for 8 hours. You are required to visit UHRS through Clickworker again.
Your UHRS earning will be transferred to Clickworker account at the end of every week. Clickworker will automatically send the money through the payout option you have chosen after holding the UHRS earning for a month.
Datamime – Powered by Tech Mahindra, Datamime is relatively new but allows instant access to UHRS. Let’s see what you need to do –
- Sign up for Datamime and provide them your Outlook ID when they ask you about secondary email address during sign up process.
- Log into your Datamime account and go to My Apps. There should be a link of UHRS login page. Click on it.
- Login UHRS with outlook ID you have given during the registration process. That’s it.
Appen – Appen is one of the biggest and best crowdsourcing website which offers UHRS access to its users. Here is how –
- Join appen(connect) and login.
- Go to all projects and Find a project that requires Microsoft live ID. Apply for the project. You will receive an email after few days containing all the information about how to get your own UHRS account.
Appen will automatically generate invoices for the work you do on UHRS weekly and payout via PayPal or Payoneer.
OneForma – Another microjobs site to gain access to UHRS. Let’s see how –
- Sign up for Oneforma account and login.
- Click on internet judging under jobs tab. Apply for job requires window live ID.
- You will receive a mail asking for your live id. Submit your live id. You will get another email with instruction about how to gain access to UHRS.
Teamwork.ai – Possibly the slowest way to get access the UHRS. Log into your teamwork account and find out if there is a job about search engine evaluation. Apply for it. They will email you if there is a position.
Also Read :
- Best Microjobs Websites to Make Money Online
- Best Survey Sites in India to Make Money from Home
- How to Start Your Own Blog and Turn Passion into Profit
Remember you don’t have to apply for UHRS with all the sites I mentioned above. Try one first. Get some experience and then you can try another. In-fact I will advise you to have only one UHRS account.
What Kind of Work/tasks UHRS Offers?
UHRS mainly offers tasks which deal with web pages for helping a search engine presenting best results to user for a query.
Types of work UHRS offers –
- Image Annotation – Like classifying images, object detection, semantic segmentation, image detection, boundary recognition etc
- Search Relevance – Identifying query-result, relevance events, entity classification etc.
- Text Annotation – Sentiment analysis like Identifying sentiment of a social media post, document classification, intent of query etc.
- Speech Transcription – Converting audio/conversation into text, relevance tasks etc.
- Video Annotation – Video relevance for a given search query, detection of objects in a video, video tracking etc.
- Other Tasks – Data collections jobs, data validation tasks, surveys, Information moderation/correction
Hope you have get an idea what you are supposed to do as a judge on UHRS.
UHRS uses some words which you may find confusing when you login for the first time. So Let’s understand UHRS terminology –
Judges – If you are completing tasks at UHRS to get paid, you are the Judge. The millions of people who help UHRS labeling data are called judges.
Marketplace – Marketplace is UHRS dashboard where you can see various categories or kind of jobs such as search relevance, image annotation, video annotation etc. You will see a number with each category in marketplace which tells how many HIT Apps in respective category are available to work on.
HIT Apps – Hit Apps are like projects. A Hit App contains tasks and all the necessary information a judge needs to know like number of tasks/hits available, earning per hit, time a hit takes to be completed, guidelines, preview etc.
HITs – Hit App contains tasks which are called hits. A task in UHRS is called HIT. No. Of available hits in a HIT App are updated in every 15 minutes.
Report – A report of all the work you have done in various HIT Apps.
Judgement Total – No. Of total tasks/HITs you have done in various HIT Apps does not matter valid or incorrect.
Judging Hours – No. Of hours you worked
Spam Accuracy – At first you might think spam accuracy has something to do with no. Of valid judgement you have made out of total judgement. When you are doing actual hits, HIT App throws testing hits to check if you are following guidelines or not while working. Spam accuracy is derived out your score in testing HITs. If your spam accuracy is low, you can be disabled temporarily or permanently from doing HITs on a HIT App.
Incentive Earning – Some HIT Apps gives bonus for best judgments. Such bonus will be displayed in reports under incentive earning.
Earning Levels – Some HIT Apps pay according to levels. Like for first few HITs you get lower pay and then higher when you enter a new level.
How to Work on UHRS and Make Money?
Login to UHRS – When you login UHRS, you see various categories according to nature of HIT Apps they contains. For an example, Text Annotation app is likely to have tasks where you will be asked something about a text like sentiment of a tweet etc. There will be a number on right bottom of each category box that tells how many HIT Apps are available to work on in that category. Click on a category that you find matching your skills.
You will see all the HIT Apps and you can get an idea about nature of tasks/HITs by reading title and description of HIT App. On right bottom of every HIT App you will see one of following 3 options –
- Train – Some HIT Apps may require training. You are required to train yourself by doing sample HITs before you encounter real HITs. When you complete a training HIT, HIT App will show a message whether your judgement was right or wrong with logical explanation.
- Qualify – After completion of training, you are required to pass qualification test by judging HITs they already know answers of. There will be a certain number of qualification attempts varies with HIT Apps. Obviously you are supposed to score good in Qualification Test. Reading guidelines and instructions really helps here.
- Start – If you see this button, you no longer require training or pass qualification and are ready to judge. However I will advise you to read guidelines before you start judging.
Start Judging – Start attempting tasks you are qualified for. Initially after a few judgements, you should go to report section to check how good you are going and spam accuracy is above the required. Each HIT App has its own rules. Some HIT Apps will disable you as soon as your spam accuracy goes below 85% while some may you allow you to continue even if it’s as low as 60%. Don’t rush through tasks. Take your time and try to be a good judge. Your reputation may bring you more tasks. HIT Apps and tasks are updated in every 15 minutes. So keep refreshing after every few minutes if you don’t see available tasks.
To avoid temporarily or permanent suspension from using a HIT App, always perform tasks you are really good at. Pay attention to guidelines. Better find other tasks relevant to your skills if you see too many red alerts during training even after reading guidelines thoroughly.
Get Paid – All that you earn doing small tasks during a week will be sent to website you have joined UHRS through which you can withdraw using one of available payout option.
How Much Money You Can Make on UHRS?
It depends upon a lot of factors like your skills, availability of hits/tasks at your location etc. I have done tasks that earned me $20 per hour and then I have participated in tasks that only made $3 per hour. You can expect a pay rate of $3 to $25 per hour. Earning per HIT depends upon difficulty level of HIT, skills required and time it takes to be completed.
How to Withdraw Your Earning From UHRS?
As you know you can only access UHRS from one of their partner website. So partner website will handle invoices and payment for you. All the money you earn on UHRS will get transferred to partner website you used to login UHRS through at the end of every week. Now it depends on payment policy of partner website how and when you receive your money in your bank account. For an example if you are working on UHRS through Clickworker, your UHRS earning will take 28 days to become payable. Clickworker will automatically process the payment if your balance is above their minimum payout threshold. On the other hand if you have gained access to UHRS via Appen, you can get weekly payout via PayPal or other payout option you have opted for. So when and how you get UHRS earning depends upon how you accessed UHRS.
Tips to Get Most Out of UHRS?
I have worked on UHRS and based on my experience, Here are few advises for you to maximize your earnings on UHRS –
Stay Logged in and keep refreshing every couple of minutes. It is important to work on HIT Apps as soon as it goes live because there will be many other judges working on same HIT App and HITs disappear fast. Judge as many as HITs as you can before you are run out of HITs or all HITs get leased out.
Read Judgement instructions carefully – Reading lengthy guidelines may seem boring and unnecessary but trust me, it not only helps you qualifying for more HIT Apps but also saves you a lot of time over long run. Some HIT Apps during qualification phase ask to judge same HITs given as examples in guidelines. In such a case it becomes very easy to qualify.
Focus on accuracy and quality rather than speed – Do not rush through HITs. Take your time. Inspect well before you judge because you never know if you are judging a real HIT or a spam HIT. Judging spam HITs incorrectly will lead your spam accuracy going below requirements and getting you disabled by HIT App eventually. On the other hand, some HIT Apps give bonus for quality judging. Once I did 25 HITs and got bonus for 7 HITs which was more than what I earned doing those HITs. If you want to stay in the game for longer, make sure you are spending enough time on reading guidelines and have a passion for quality and accuracy.
Don’t Get Disabled on HIT Apps – A Hit App is like stream of money. New HITs keep coming regularly in an HIT app. So you always have chance to earn money regularly until you get disabled permanently from the HIT App. When you don’t pay attention to instructions and keep doing HITs incorrectly, you get disabled temporarily. Consider this a serious warning. You only get 2 such chances before you get disabled permanently from doing HITs on a HIT app and thus ending up losing a stream of money forever.
Work on Laptop/PC instead of Mobile Phone – Laptop makes research and multitasking easy. You can open a lot of tabs for research and read instructions properly.
Multiple UHRS Accounts – As you already know, you need to sign up for a company/contractor to get access to UHRS. Some companies don’t mind if you are already accessing UHRS through another company. So you can have multiple UHRS accounts. I will advise you to use different browsers/devices for using different UHRS accounts. There are chances that you will have different HIT Apps in different UHRS accounts based on your profile information you mentioned while signing up for UHRS partner website.
This was all for this post. Hope you found it useful.