It was a sunny March afternoon when I walked into the parking lot of the swanky Waters Edge Hotel in Colombo. SocialCops’ co-founder Varun and I had just given a presentation amongst a group of policy-makers, civil society members and decision-makers at a #DataForSDGs conference held by the United Nations Sri Lanka. We had discussed the data gaps plaguing the decision ecosystem in Sri Lanka, and how our SDG Tracker could help solve them and set up the country for success in achieving the ambitious Sustainable Development Goals (SDGs).
I’m not going to lie — it was a bit bizarre. Just a year ago, I was just a college student in Boston eating mozzarella sticks and worrying if I could submit my finals papers on time. How did I end up addressing decision-makers in a foreign land, guiding them on how to best achieve something as important as the SDGs, in just 1 year?
It was simply because of SocialCops. During the taxi ride back to my hotel, I reflected back on the amazing journey I had as a Data Analyst at SocialCops, which started in August 2017 in Delhi and took me all the way to Colombo.
I started my journey as a Data Analyst in the Research & Analytics Team at SocialCops. The R&A Team consisted of folks from backgrounds as varied and unique (development economics, data science, research, policy and governance, and more) as the problems the company was trying to solve.
To be honest, I joined the company not knowing what I wanted to do exactly. I had some sense of technology, analytics and economics from courses in college. I had some consulting experience from an internship at a Big 4 consulting firm. I was a beginner in R, after taking a single analytics class in college and using that basic knowledge during a part-time internship at a Boston-based startup. I wasn’t sure exactly where I could put my learnings into action. But I definitely did know that it would have to be driven by my impetus to make big impact using technology.
Read more about why I joined SocialCops in the first place.
The DISHA Direction
I started working on DISHA — a landmark data-driven governance and policy initiative to onboard 42 national flagship schemes onto one platform, led by the Government of India. I was a scheme-onboarder, meaning that I helped bring the scheme data into our platform.
This was my very first tryst dealing with data at scale, and what a tryst that was. I was responsible for the entire lifecycle of a scheme’s data, which meant I took on a lot of tasks:
- Conducting scheme research to learn about the schemes
- Coordinating with ministries to provide the data via web services
- Conducting data quality checks to ensure the sanctity of data streams
- Writing R scripts to automate the data cleaning process
- Building visualizations on our internal dashboard platform Visualize
- Onboarding the schemes onto our internal pipelines to automatically update the data regularly
- Deriving insights from the visualizations to help spark conversations and decisions among decision makers
I learnt so much about the way the Government of India functions and how its schemes are administered to provide last-mile benefits to beneficiaries. For example, did you know that under the National Rural Livelihoods Mission (NRLM), over 50 million rural households have been initiated into Self-Help Groups that are meant to foster self-employment and entrepreneurship? Now that is scale!
My learnings were not limited to just knowing so much about the government’s efforts towards development. I also grew as a data analyst. I had never written a production line script in R ever, and now I was writing code that would plug into data pipelines and clean data on a regular basis — the kind of work that a bunch of data scientists/engineers would do in other data science teams. DISHA was also the first time I was exposed to GitHub and SocialCops’ focus on peer-reviewed code.
A major highlight of my time with SocialCops and DISHA has to be watching the Prime Minister launch the dashboard in front of us!
It was surreal to sit in the audience and watch a video highlight screens that I had contributed to, showcased to none other than the Prime Minister of the country. How many 22 year olds get to say that?
In addition to onboarding schemes onto the platform, I also helped create DISHA Analytics — a layer of data intelligence on top of the real-time scheme data to intelligently help Minister of Parliaments figure out trends, identify areas of growth, and pinpoint opportunities for improvement across all schemes and geographies in just a few clicks.
It’s a testament to the data culture at SocialCops that, before joining the company, I had never written a production line R script, and in just a few months I was building a crazy pipeline across 18 schemes for data analytics that looked like this:
DISHA was a crazy and memorable ride that I won’t ever forget. Here are some numbers from my time with DISHA:
- 5 schemes and analytics onboarded
- 15 trips to government offices
- 17 dashboard screens built
- 31 R scripts written
- 40+ phone calls to ministry officials
- 129 Github commits
- 446,387 lines of code written
Scaling SDGs
Along with working on DISHA, I was assigned to internally manage the SDGs project at SocialCops. While DISHA was a journey full of government office trips, phone calls to NIC and checks to ensure live data passed through our pipelines seamlessly, SDGs was a journey of building our product elements to power our SDG solution (a platform to help countries monitor their progress toward the SDGs and Agenda 2030).
Check out Anjori’s story of what it was like to be an Account Manager on the SDG solution.
I first started off with building the SDG dashboard for India, which used data from over 100 data sources across the 200 indicators. As part of this, I was also working on building our internal data repository of data — scraping PDFs and website tables, cleaning them and structuring them for analysis and visualization across India’s national SDG indicators.
This was another opportunity for me to build my skills as a data analyst. I worked in data cleaning primarily in R. To make this easier, we have actually built an internal R library of functions called “CleaningHacks” that makes it easier to clean data. For example, we have a function called ‘StandardColnames’ that converts all dataframe columns into an internally consistent, analysis-friendly format. These functions really sped up my process of cleaning the data. Towards the end of the project, my comfort level with R functions grew so much that I actually contributed 3 functions of my own to the CleaningHacks library.
At the end of the project, the SDGs India dashboard had:
- Over 900 widgets
- 202 indicators
- 130+ data sets
- 6 different tabs and views
It wasn’t just about building the dashboard, though. As part of our work, I also traveled around India to present the dashboard to various stakeholders, including various state governments and UN volunteers.
Off to Sri Lanka
As an offshoot of our work in India, we were approached by the UNDP Sri Lanka to help them build their national SDG tracker. We built the first prototype of this dashboard in collaboration with CitraLab, Sri Lanka’s first Social Innovation Lab, and the WHO. This prototype visualized 31 indicators for Goal 3: Good Health and Well Being. It’s actually publicly accessible right now, so you can check it out here: unsdglk.socialcops.com
The cool thing about working with a lot of data is that once you get a hang of the process, you can scale your work really quickly. It took us about a month to finish just one goal on the SDGs India dashboard. Then, after we set data sharing standards and scripts based on our learnings, we were able to push live the first version of this prototype in just two weeks!
As part of the launch of this dashboard, SocialCops was also invited to speak at #SDGDataLK, Sri Lanka’s first national symposium on data for the SDGs. So SocialCops’ founders (Prukalpa and Varun) and I boarded a plane to Colombo to present the dashboard to various stakeholders in Sri Lanka.
This brings us to the presentation I talked about at the beginning of the post. It was an amazing experience discussing the actual data gaps that prevented effective decision-making in Sri Lanka.
Moving from “Why?” to “What’s Next?”
During my 1 year as a Data Analyst at SocialCops, DISHA and SDGs were just two major things I got to work on. But I also played a part in…
- Conducting primary research on Papua New Guinea’s development while we deployed our SDG solution in the country
- Building a dashboard to track toilet constructions for a project in Uttarakhand by the Hans Foundation
- Giving sales demos of our SDG solution to 5 different state and country-level groups
- Building a financial analytics dashboard for the Assam Finance Department to help them allocate state resources better
This incredible journey at SocialCops started with a simple one-word question — “Why?” This question popped up during my Big 4 consulting internship during college. I knew that no matter what I did, my work had to not only speak to that question, but also satisfactorily answer it.
Today, just a year after joining SocialCops, my work has not just spoken to that question, but added countless memorable moments with the amazing people here and exponentially honed my skills as a data analyst. Now, the only question that pops up in my mind when I look at my work is “What’s next?”
