The platform has seen huge success which resulted in a lot of performance issues around serving the dynamic volunteering tasks. Amnesty realized in order to increase the performance of the platform they would have to redesign the engine that managed the tasking.
What is Microtasking: https://en.wikipedia.org/wiki/Microwork
When reviewing the issues with the Amnesty Decoder platform it was clear that the core issue was due to the database being under heavy load from all the requests for assets and collection of data. They were using an off-the-shelf product that wasn’t design for such a large scale, and running it on a generic LAMP stack with no scaling capability.
Our solution was to take their existing components, turn them into services we could scale within the cloud and then create a bespoke API that could act as the engine to run the tasking and authentication.
Amnesty was already using Azure for deployment of their current systems. So they had the ability to scale, but didn’t possess a orchestration system to scale automatically. With the help of their internal DevOps team we took their existing components and containerized them while adding some additional services to manage shared caches.
This then allowed Amnesty to automatically add more nodes to the cluster to scale the services like MySQL, PHP or Nginx.
It was very important that tasking system would collect and save the data quickly as possible. Our initial review of the off-the-shelf tasking system showed we would still suffer performance issues with scale even if we scaled the resources. Because of this problem we created a brand new bespoke API which was designed to solve the issue. This new API wrapped a open-source project called Hive which used ElasticSearch as a database with tasking logic written in GoLang for speed. This would then pass through a PHP API allowing the addition of authentication using Oauth 2.0.
Our changes resulted in a system that could support high numbers of requests and would horizontal scale with more nodes. Giving Amnesty a solid base to continue running digital crowd volunteering into the future.
Amnesty managed to run their next campaign with 20,000 users and close to 250,000 tasks being complete.