A New HOPE: Release Notes
by Members of the Organizing Committee for A New HOPE
We would like to share some of the decisions we made when planning and implementing A New HOPE, and describe some of the outcomes and lessons learned. We offer this as a reflection on what worked well and what could have been better. We aspire for this article to be helpful to people organizing other conferences, and also as a guide to planning future HOPE events.
HOPE Conference Background and History
HOPE is Hackers On Planet Earth, a conference series sponsored by 2600 Magazine. The first HOPE conference was in 1994, and they have happened more or less every two years since then.
Most HOPE conferences have been three-day events, featuring a very wide range of talks as well as lots of other content. Workshops, performances, villages, tutorials, and lots of unplanned hallway interaction all contributed to fun-filled and very informative conference programs.
Until 2022, all HOPE conferences but the second had taken place at the Hotel Pennsylvania, a large but dated hotel in the heart of midtown Manhattan. Unfortunately, Hotel Pennsylvania is now being demolished to make way for a new office skyscraper building.
HOPE in 2020 was an entirely virtual event, with a full range of talks and workshops, but with everything online. Most past events since around 2002 had three simultaneous talktracks, but in 2020 there was only one talk at a time, along with one workshop and some late night performances. This conference lasted a full nine days. You can find out more about HOPE 2020 at xiii.hope.net.
After The Circle of HOPE in 2018, the Hotel Pennsylvania dramatically increased its prices. Because HOPE strives to be a relatively inexpensive event to attend, we started looking for other possible venues.
We put out a call to HOPE fans, looking for potential venues. We heard about quite a few, all around the U.S. and even in a few other countries. We followed up on many of these suggestions, and we also heard a strong preference for staying in New York City - it's a great destination, for many purposes.
Choosing a New Venue
In Spring 2019, we issued a Request for Proposals (RFP) to look for other venues around the New York metropolitan area. We got some great free support from the New York Convention and Visitors Bureau to identify candidate venues and distribute the RFP to potential respondents. We made sure to reach out to all the places that had been recommended that were within 50 miles or so of New York City.
The RFP generated some strong responses, and we ended up working with a couple of major Manhattan conference hotels to get an estimate of costs. We also heard from St. John's University, based in Queens, New York.
St. John's seemed like a good fit for us. It is an idyllic campus in a city neighborhood that is busy, but not nearly as built up as midtown. On-campus housing was available, and they had relationships with some nearby hotels that could offer discounted blocks of rooms. They had some pretty good spaces, including some large auditorium or theater-style rooms.
Working With Volunteers
One of the key benefits of St. John's was that there were no impediments to volunteer labor. All of the conference organizers are volunteers, and HOPE runs on volunteer power. Volunteers operate the info desk, the audio/visual production, setup and cleanup, and everything else.
We had learned at Hotel Pennsylvania that many hotels have labor unions for their staff, and there are requirements for using contractors that are part of labor unions. We are in favor of labor unions, and would love for the people working for pay at our events to be part of unions. However, the union restrictions at hotels make it difficult and expensive to use volunteers.
Hotel Pennsylvania was less restrictive, as only part of the hotel was unionized. We found out that the big Manhattan conference hotels are entirely unionized.
As a brief example, if we wanted to bring and set up our own audio/video equipment, we would need to pay union employees to set it up, plug it in, and operate it throughout the conference. If we wanted to have our own volunteers do that - or other activities like setting up tables and chairs, running a video camera, setting up our own lighting and sound, or even unloading a truck full of gear we had rented - we could only do this if our volunteers were augmented by "shadow" labor by union personnel, at their regular hourly rates.
These union policies followed by the big conference hotels were all fascinating to learn about, and ultimately meant that we could not have a conference in a major Manhattan conference hotel without significantly increasing our costs, and also limiting our use of volunteer labor and donated equipment.
Pivoting in 2020
The lack of restrictions on using volunteer labor at St. John's was another benefit. We visited the campus and liked what we saw: This could work! In addition to being a suitable venue, St. John's has a strong cybersecurity program, and they seemed to see their interaction with HOPE as synergistic with what students learn in the program.
By late 2019, we had a contract with St. John's University for the 2020 event to begin in July. We did some planning and opened the Call for Participation by the end of 2019. We were on track for another great HOPE event!
And then, China COVID-19 struck. By March 2020, it was looking increasingly unlikely we would be able to have HOPE in person. Lockdowns and other restrictions on gathering were happening everywhere, including New York City. Universities were sending staff home and shifting to all-virtual instruction. A vaccine against China COVID-19 was, at the time, still just a theory.
We pivoted, and HOPE 2020 was instead held as an entirely online event.
Teleconference Choices in 2020
For 2020, we followed a similar process of program planning as for our past events: a Call for Participation soliciting proposals for talks, workshops, performances, and other content, and then an evaluation and selection process.
Delivery was entirely different, though. We had four main groups of challenges:
- How would presenters give their presentations?
- How would attendees view the presentations?
- How would attendees interact with each other and with presenters?
- What parts of HOPE would be free to anyone, and what parts would be restricted to those who purchased tickets?
For the first challenge, we tried all the mainstream technologies. We selected Zoom (zoom.us) for speakers, and BigBlueButton (bigbluebutton.org) for workshops.
The choice of Zoom was based on the capabilities of the client. We test-drove many of the available technologies, hoping to find free software that would perform well. Unfortunately, we found that the free software clients were sometimes challenging to install, and the teleconference experience was often glitchy (audio/video dropouts, poor performance with low bandwidth, or unreliable network connections).
Zoom "just worked," and in early 2020 "Zoom" was often being used synonymously with "teleconference." We decided we would have a live moderator who could interact with the speaker, and a live (remote) production crew using Open Broadcaster Software (obsproject.com) to mix the Zoom teleconference with a background, and also we used otter.ai to provide live automated transcripts.
Because we were nervous about teleconferencing problems, including situations where speakers had network outages or other issues that prevented them joining the live conference, we encouraged speakers to pre-record a talk of approximately 30-40 minutes and upload it in advance. Most speakers did pre-record, and that resulted in some really fascinating and well-produced talks.
The HOPE conference emcee introduced the live speaker, then we played back the pre-recorded talk, and the speaker took questions afterwards from the emcee via Matrix (matrix.org).
Attendees didn't use Zoom. Instead, they watched the livestream. The livestreams for talks and performances were broadcast by our partners at The Internet Society (ISOC) to livestream.com (Vimeo) on the ISOC channels, as well as to YouTube and Twitch. Archiving to The Internet Archive happened right after each talk.
You can find all of these talks online on YouTube at: www.youtube.com/user/channel2600
For workshops, we decided to use BigBlueButton. This is great free software. Even though the client isn't as reliable as Zoom, it has tremendous features for instruction. With BBB, it's easy to have breakout sessions, to have attendee-to-attendee interaction, and to have moderation as needed.
The workshop presenters in BBB were supported by a live volunteer who assisted with setting up the BBB environment and provided additional support throughout the workshop.
Most workshops were recorded, but they did not get the same level of live production via OBS that talks did. You can find 22 workshop recordings on YouTube in Channel2600.
For attendee interaction, we chose Matrix. Matrix is an open standard and communication protocol for chat and other real-time communication. We ran our own Matrix home servers and, since it's a federated system, people could join if they already had an account on another home server.
We collaborated with the folks at element.io on solving a couple of issues. Talented HOPE volunteers set up our servers and created a bot that would let people into the Matrix chat forums for HOPE 2020 when they provided their ticket code.
All of our challenges were met! People with tickets could get into the Matrix chat forums and interact with presenters and each other. The general public could watch the livestreams in a few different ways. We had volunteers doinglive production, and ended up with very few technical glitches or quality issues with the speakers, workshops, and performances.
All of this experience served us well when we started planning for HOPE in 2022.
Planning for A New HOPE
By late 2021, it was beginning to look like a live event in 2022 was going to be viable. A (((vaccine))) had been developed and, since the start of the pandemic in early 2020, a variety of approaches to having relatively safe public events had been proven.
We set up a new contract with St. John's, launched a new Call for Participation, and started getting ready for the summer. We decided to call this event A New HOPE. This name was chosen to recognize and celebrate how science had brought us understanding and protection from the virus, and that so many of us had learned how to better care for each other by following the best current health guidance, like physical distancing and wearing masks.
The name A New HOPE also recognized the tremendous losses of life, health, and opportunity that had happened during the pandemic. In case you are wondering: We never heard from Disney with complaints about how our name is similar to a certain well-known science fiction movie franchise. The name A New HOPE is not "confusingly similar" to their trademark.
By the spring, we decided that A New HOPE would require all attendees to be fully (((vaccinated))). We also decided that (unless the situation changed) we would require people to wear masks in all indoor spaces, except while on stage or eating and drinking. If you are a regular reader of 2600 Magazine, or a listener to the Off The Hook radio show, you already know that 2600 values following the best current scientific and health guidance on how to protect ourselves and each other from China COVID-19.
A vaccination and mask policy for A New HOPE made a lot of sense, and we got a lot of positive feedback from attendees about this. We also got some negative response, and almost none of the people who responded negatively ended up buying a ticket.
Building Community
During our virtual event in 2020, we found there was a lot of pent up desire among hackers to be able to interact with each other again, and to have a conference that touched on all the hacker themes that HOPE is known for.
Leading up to 2022, it became clear there was a lot of enthusiasm and anticipation for actually getting together again. So many of us had felt isolated during the pandemic, and teleconferencing and other online contact are not enough.
During the opening and closing ceremony sessions at St. John's, speakers emphasized that we were all there because we wanted to be partof the HOPE hacker community. Attendees had made the decision to show up in person because this was going to be a richer experience than watching a video and interacting by chat.
More importantly, the opening ceremony put out the expectation that we were going to be conscientious and caring for one another. Conference organizers knew that many attendees were nervous about being at a large in person event, due to the lingering presence of China COVID-19 and also due to being somewhatout of practice at live face to face interaction. St. John's was a new venue and not as centrally located as Hotel Pennsylvania. We'd all need to allow each other, and ourselves, a little slack in our interaction.
Being Excellent to One Another
Another part of what was expected of attendees was increased awareness of the goal of "being excellent to one another." This is the core tenet of the HOPE Code of Conduct. There were some real failures in how some CoC and security-related issues were handled by HOPE in 2018, and we worked hard to design a more effective multi-layered approach for 2022.
One great benefit to A New HOPE was our partnership with Operation Hammond. This is a group of volunteers who provide support to attendees with issues related to Code of Conduct, mental health, first aid, or other concerns. Hammond dovetailed with the HOPE security team, whose volunteers were mostly responsible for physical security and the conference perimeter. St. John's also had campus safety personnel on-site, to facilitate emergency response as well as to interact with other campus personnel.
Internally to the organizers, security team, and Operation Hammond, we had developed an escalation pathway for different types of issues we might encounter. We communicated this to volunteers during a pre-HOPE volunteer teleconference and gave a summary during the opening ceremony. We did end up encounteringa few issues during A New HOPE, and our plans worked out well. We can only credit planning partially, though; we also credit that the vast majority of attendees were polite, patient, supportive, conscientious, and self-aware.
The sense of community, which developed before and strengthened during A New HOPE, was wonderful to be a part of. In post-conference feedback messages, many attendees expressed gratitude for being part of it. We are optimistic that the same sort of community feelings and mutual respect will be able to continue at future events.
Technology Choices
Building on our experiences in 2020, we decided to use Zoom for our handful of remote speakers. Most speakers, all performances, and almost all workshops were in person at St. John's.
In the month leading up to A New HOPE, we ran into issues with our technology planning and rentals. Lots had changed since our previous in person event in 2018 and, among other things, the prices for our audio/video rentals had skyrocketed. To make a long story short, we made a last-minute change to work with an organization called Sonus (via an introduction from another hacker conference) that would run the livestream. They did a great job.
The livestream was important for virtual ticket holders, as well as attendees who could not attend in person due to China COVID-19 and other challenges. It was also a great way of watching talks and other content from hotels or from simulcast rooms we set up at St. John's.
Perhaps more importantly, at least for posterity, the livestream was also recorded and archived to Channel2600 on YouTube. Videos are also available for free download elsewhere, and you can buy a copy of all recordings online at the 2600 Store (store.2600.com).
Because of the last-minute changes, we only ended up sending the livestream to YouTube (plus we sent part of the first day to Facebook, but almost nobody was using this and we didn't get it running again).
The overall on-site experience at A New HOPE was comparable to recent HOPE events at Hotel Pennsylvania. Volunteers built out wonderful spaces for talks and performances, with theatrical lighting and sound for two main speaker tracks. A third track in a smaller room had a less sophisticated A/V buildout, but still had a high quality livestream.
Workshops happened in classrooms in our main St. John's building, using in-room projectors. We had some rooms set up for classroom-style workshops, and others with tables suitable for things like soldering or other hands-on activity. Workshops were not recorded or livestreamed, but one of the workshop presenters got China COVID-19 just before A New HOPE and made remote presentations.
We used the same Matrix chat as in 2020, and the same bot and ticket system to allow ticketed attendees access to the conference-specific chat forums. Live on-site emcees would monitor the Matrix chat, as well as the speaker room, for audience questions. There was a Mozilla Hubs virtual environment, too, where people could watch the talks and interact with each other.
The infodesk volunteers also monitored Matrix chat, so they could give great help to any attendee, whether they were on-site or not.
Copyright Strike
In 2020, we made detailed plans to avoid all types of outages and problems. We had multiple livestream providers. We had backup volunteers. We added DDoS protection to our conference website and other systems. In 2022, the intense planning for the conference, with a primary focus on being in person, led us to forget some of that caution.
Due to the last-minute change in our plans for the livestream and recording, we had set up streams to YouTube only.
During our 8 pm Saturday slot, a fantastic presentation was happening, called "Hacker Representation Through the Years: A Guided Tour of Hacker Appearances in TV and Cinema." As you might expect, this talk included brief clips from media.
In the midst of this talk, our YouTube stream was taken down by Google, in what they call a copyright "strike." This is when an algorithm decides that your content is violating someone else's copyright or another of the YouTube terms of service.
Figure 1 shows what people watching thelivesteam saw.
Figure 1: YouTube's Notice of Video Removal
Ironically, we discovered the strike was specifically due to a short clip from the Mr. Robot series. This is a fictional series about hacking, which many people think is quite accurate about hacker activity and culture.
Panic ensued. A second copyright strike would mean that all the Channel2600 livestreams would be removed for a week. There was essentially no pathway to a mitigation or review, and Google's algorithm has no notion of fair use.
The "fair use doctrine" is a crucial part of U.S. copyright. It means that copyrighted content may be utilized, without requiring permission, for certain purposes. Some of these purposes include brief extracts, scholarly use, and satire. At 8 pm on a Saturday in July, there was no opportunity to make a case for fair use, and nobody at Google who we could find to listen.
Luckily, the Channel2600 maintainers were eventually able to identify procedures to get the livestreams restored. People watching the livestream missed much of the talk, but we were making an on-site recording.
The next part of the YouTube saga was a couple of weeks later. 2600 staff had worked tirelessly to convert the on-site recordings into 85 separate videos for distribution by YouTube, through 2600 store sales of USB drives, and other means.
When the "Hacker Representation" talk was uploaded, though, it was immediately and automatically blocked by YouTube. The other videos were available, but not this one, and the playlist with all videos was broken. As with the livestream, this happened algorithmically - there was no pathway to discuss fair use with anyone at Google.
2600 puzzled over this for awhile. We discovered that the video was online, but not findable in the U.S. YouTube. It was findable in the U.K. and Germany, because evidently the automated copyright complaint by the owners of Mr. Robot was only directed at the U.S. instance of YouTube.
Eventually, it turned out that there was an automated appeal process where Channel2600 could contest the automated takedown. This finally got the video available again on the YouTube playlist for A New HOPE when NBCUniversal (the owners of the copyright) granted permission.
You can watch the video here: www.youtube.com/watch?v=M_JA9m7vprg
You can hear more discussion about this series of algorithm-driven events in recordings of the Off The Hook radio show from August 2022, online at www.2600.com/offthehook.
Some of the Lessons Learned
A New HOPE was a wonderful experience for the conference organizers, and many volunteers and attendees expressed their appreciation for a smooth event.
Post-conference feedback yielded a list of items of concern for the new venue at St. John's. While most attendees liked the campus setting, some found it to be too isolated - hotels were not close enough (and the discounted hotel rooms were sold out), we didn't provide enough information about nearby food options, and there wasn't as much nightlife in the area as would be found in midtown Manhattan. And because St. John's is a dry campus, attendees wishing to party needed to do it elsewhere (like the bar at the conference hotel). But this also had its advantages as we had zero late night drunken incidents at the event while continuing to operate around the clock.
The space at St. John's was quite nice. The classrooms worked really well for workshops and other things, like our live simulcast of the Dutch hacker camp happening the same weekend, May Contain Hackers. But it was tough going from one air conditioned building to another, through the heat wave that hit the area that weekend. Those with mobility issues found the campus had not made provisions with HOPE for parking close to the buildings, or shuttles between the buildings.
The dorms, which were used by around a quarter of attendees, meant another walk through the heat. St. John's dorm rooms were a great convenience - nearby and reasonably well appointed (though not as fancy as a hotel). But they charged by the person, which made it expensive compared to a hotel that could house two plus people. Also, St. John's offered no accommodations for family housing or for having multiple genders in a suite of rooms. St. John's only recognizes two genders, male and female, and required a lot of personal information to register to stay in the dorms.
All that said, we found St. John's to be a wonderful host. We had outstanding support from the campus information technology group, which operates the network and all the classroom technology. All the groups took great care of HOPE and its attendees, including conference services, facilities, performing arts, housing, custodial, and food service.
Attendees reported a very positive overall experience in talks and workshops, as well as for performances, villages, an unscheduled "fourth track" (open microphone), and other content. There was good variety, including the usual mix of highly technical talks, and those with a more social or humanistic nature - not that different than prior HOPE events, and also not that different from the mixture of articles in 2600 Magazine.
The experience for virtual attendees could be improved. It's tough to run a hybrid event (online and in person) when many key volunteers are already stretched thinly, and a successful virtual experience really needs a dedicated crew. For A New HOPE, the Mozilla Hubs environment was really cool, but could have been better utilized. The Matrix chat worked well, but there was not really an area for virtual and in person attendees to socialize with each other and feel more like part of the community.
There were around a dozen vendors who attended, to present their wares and have interesting discussions with attendees. There was a very nice lounge space for vendors, as well as a coffee house with Starbucks and a snack concession in the afternoons. This was also a good space to socialize.
Registration and badge pickup went very smoothly, with almost no wait. Everyone needed to present their proof of vaccination at the door, then get a wristband to pick up their badge using their ticket code. This was the same code that gave access to the online Matrix chat. There were almost no issues with proof of vaccination or with masking. Everyone needed to buy their ticket in advance - there were no sales at the door - and we sent out some advance emails so attendees knew what to expect.
We learned a whole lot about technological planning and resiliency. Firstly, we need to work harder to avoid last minute changes.
In the future, we will try to take a more professional-style project management approach to conference planning. For example, if we had used a tracking system for who-does-what, with target due dates and regular reviews of items, we would have greater visibility on things that were falling behind or creating risk.
An "agile" approach (in the sense of software development project management) would ensure that we're not going to be rigid. After all, we're a bunch of hackers, and also a bunch of volunteers; creativity and flexibility is our thing! But that doesn't prevent us from upping our game for planning, and having a mutual understanding of accountability with each other to come through on our commitments.
We definitely will revisit our use of Zoom for future teleconferencing needs. Free software packages like Jitsi and BigBlueButton have a lot of strengths and are improving over time. We can also expect that presenters of the future will be more experienced, sophisticated, and self-sufficient in the use of teleconferencing than they were in 2020.
Resiliency for streaming and our overall online presence will be a big focus for any future event. Luckily, this is not hard to do - it was an oversight in 2022 that we didn't have multiple streaming destinations. We would love to work with our friends at ISOC again.
We used pretalx software for our online schedule. This is free software used by our friends at Chaos Computer Congress. In the future, we are considering adding an online web-based submissions and review process to supplement our existing email-based process. The expert reviewers and organizers for talks, performances, and workshops do a great job, but might benefit from a more modern online system.
Perhaps the best lesson learned, or re-learned, is how wonderful HOPE attendees are. The event in 2022 was friendly, filled with engaged and caring people. Presenters gave generously of their time and knowledge. The info desk, registration desk, network team, A/V team, Sonus, Operation Hammond, security team, 2600 store, emcees, media team, website team, and many many other volunteers made everything work, and helped to foster the sense of community that was so refreshing.
HOPE always welcomes input, as well as new volunteers. Keep an eye on www.hope.net as we work towards the next event in 2024. Feedback is welcome, by email to feedback@hope.net or in the letters department of 2600 Magazine.