2015

Fixing hackathon inefficiencies

Becoming more efficient at supporting Devpost’s hackathon offering

Problems to solve

Devpost wanted to shift its business focus to other areas besides hackathons, but support for these events consumed much of the team’s resources. Many of these issues stemmed from hackathon managers’ confusion about our product, resulting in a subpar experience for participants.

In 2015 Q3, we spent an estimated 430–525 hours on hackathon support in areas that could be addressed by improvements to the product. So we used our OKR quarterly goal system to prioritize making our product more self-serve.

Goals

  1. Reduce Devpost team member time spent supporting hackathons
  2. Address managers’ most common requests

Planning

To create the biggest impact, I identified which issues consumed the most time/resources by interviewing members of our tech, business development, and marketing teams. We discussed their recurring pain points, and estimated how much time they spent dealing with each issue.

Once common issues were grouped by theme and ranked by time spent, I pitched high level solutions to our tech lead to get a rough idea of technical effort so I wouldn’t over-commit our team during the time allotted.

I used this system to select the top 6 pain points as my product goals for the upcoming quarter.

Solutions

1. Help participants submit their project to the right place

Despite improvements to the project submission flow, each week a few users would mistakenly submit a project to their portfolio, and not the hackathon. This created a big problem for managers, who had no way to relocate the project to the right place.

We attacked this problem in 2 ways: First, we created a feature allowing managers to accept late submissions if a user missed the deadline. Occasionally this happened at chaotic in-person hackathons, and empowering the manager to re-open the submission period for a specific user provided a powerful tool at the most stressful point of the event.

Late submissions toggle

Second, a new modal popup now appeared when a user added a project to their portfolio, confirming whether they were submitting to their portfolio or to a hackathon. If they indicated they were submitting to a hackathon, the modal displayed a list of open hackathons for the user to choose from, then directed them to the proper submission flow. After releasing this feature, we stopped receiving any support issues regarding improperly placed projects.

Modal popup confirming whether a user is submitting to a hackathon or their portfolio

2. Automate emails

Devpost employees in charge of in-person hackathons sent A LOT of emails to hackathon managers, encouraging them to get their site published early and offering tips along the way. Getting a hackathon site published early was important so that participants could register prior to the event and take advantage of features such as team building and discussion forums.

While the emails were effective, they had to be sent manually – a huge pain for our team. To improve this workflow, we used our favorite email tool to automate 5 emails at key points throughout an in-person hackathon. The emails were still written from a personal point of view, and if a manager replied they’d reach a real person. This solution made a big dent in the amount of manual email work for our team.

Automated emails reminding managers to publish their hackathon (left) or announce the winners (right)

3. Self-publishing

Managers could start a draft hackathon on their own, but only a member of our team could publish it and make it live on our site. This helped us maintain high quality content, but was not a scaleable solution. Additionally, this workflow wasn’t obvious to managers, so we received a lot of “quick, I need my hackathon published now!” messages – especially inconvenient because a typical hackathon occurs over the weekend.

As a first step, I updated the design of the hackathon management area to make it more apparent when a hackathon was in draft mode.

Draft mode of an unpublished hackathon, before (top) and after (bottom)

Next, we opened the publish option to managers. To clarify what type of content was acceptable, the first step of creating a new hackathon displayed a checklist. These guidelines appeared again when a manager published their hackathon. They had to agree to each item before the “publish” button was enabled.

Publishing guidelines checklist

To avoid lowering our content standards (or worse, flooding our site with spam), we created a checkpoint after a hackathon was published. A member of our team must manually approve a hackathon before it appears in our hackathon listings or newsletter, but the page itself would be public. Managers could use the live link without needing our approval, but we’d still control what content was shared with our users. Any hackathon that violated our terms of use would be removed.

Hackathon listings page

4. Custom submission questions

Hackathon managers frequently requested adding custom questions to their submission form to gather data unique to their event (e.g., applying for sponsor prizes, submitting within a specific category, etc). We gladly offered this option, but the feature was only accessible by Devpost employees. Aside from an awful UI, the underlying tech was very unstable. Using the feature “the wrong way” could break the entire submission form. We hoped to open this feature for managers to use, but first it needed a serious overhaul.

Our tech team rebuilt the entire custom submission question form builder. They accounted for the cases we often ran into, such as a manager wanting to add/edit/remove a question after the hackathon had already started. They also removed the unnecessary options that had piled up over time, simplifying the interface to its essentials. Lastly, I redesigned the UI to make it more user friendly and match the visual esthetic used elsewhere.

(left) Existing custom submission form builder – world's most painful UI.
(right) Wireframe for integrating the improved feature into the hackathon manage area, including a consistent UI for the various options.

5. Judging

Devpost was built for online competitions, so the judging process also took place on the web. But this conflicted with what occurred at in-person hackathons. These events typically used science fair style judging (a judge walks around the room and scores projects one by one) or presentation style judging (the participant presents their project on stage to a panel of judges). Using either of these methods, our online judging system was more of a hassle than a help.

So I designed a way for managers to “skip” online judging if they didn’t need it. The platform now presented 2 options to managers: online judging, which maintained our existing features, or offline judging, which removed fields such as judging period dates, judges’ email addresses, and the requirement to moderate submissions. This was data our system didn’t need if judging was handled offline, and made the hackathon setup process much faster for managers.

Judging options change based on online or offline mode selected

6. Updates

Every hackathon on Devpost has an “updates” feature that helps managers keep in touch with participants. It’s part blog and part email newsletter.

Our marketing team manages updates for our full-service clients, so I used their input to improve the update experience. Changes included:

Old updates interface

Improved updates interface

Lessons learned

Rank internal requests by time spent per quarter

Everyone has a passion project they want the tech team to fix. I’m certainly guilty of lobbying for features that would make my life easier! For this project, it was an eye-opening experiment for employees to quantify pain points by hours currently spent on the issue, and gave me an impartial method to prioritize work for the quarter.

Bring everyone to the table

Product managers act as the bridge between tech and business, relaying information between both parties. Fixing inefficiencies isn’t as exciting as building a shiny new feature, so I motivated the tech team by helping them empathize with the pain our colleagues and hackathon managers were dealing with.

For each project, I led a kickoff meeting with a tech lead and relevant business/marketing folks. We demo’d and fully documented the existing problem, then noted how we dealt with it (which was usually cringe worthy from a tech perspective). Next we brainstormed various solutions, from quick fixes to overhauling entire sections of the platform.

Simply getting everyone in a room at once resulted in collaborative brainstorming, eliminated rounds of emails typically needed to get everyone on the same page, and energized the whole team about working towards a solution together.