Designing LinkNYC for NYC

by

Value Sort_Martin (1)

As Fast Company reported last week, Control Group is in the process of conducting ethnographic research to understand exactly how people across NYC might experience LinkNYC. As partner Colin O’Donnell explained: “We want to reflect the diversity and interests of the city and fit in aesthetically and functionally with what people want.”

UserInterview_Mariama (1)How do New Yorkers perceive things that are “free”? How is video chatting best communicated? What are different public screen scenarios that people are used to? This important human-centered design phase will help inform our entire development process here at Control Group – from user experience design, to interface design, to software development. In this way, we are not only able to build strong community partnerships to learn more about citizens’ needs across NYC, but we are also able to communicate directly to these neighborhoods about Link simultaneously. This will ultimately help foster both widespread adoption and usability.

FocusGroupBrooklyn (1)We’d like to thank the following organizations who have welcomed us to their neighborhoods and provided much needed insight and guidance on how their constituents would like to experience Link in New York City:

Outreach_siliconharlem (1)And the ones we will visit in the coming weeks:

Of course, this is just the beginning. Just as New York and New Yorkers will continue to evolve over the next decade, so will Links. And this type of research and engagement with different communities will help inform what Link will do, whether in 2015 or 2025.

Unit testing Websockets in Play 2.2.x

by

150307 Yang_EMKI Surround-113On a recent project for The Edward M. Kennedy Institute, we made heavy use of the Play framework and its underlying Akka implementation. As part of the system architecture, we applied the concept of entities from Domain Driven Design to that of an actor. This idea was sourced from Jamie Allen’s book, Effective Akka. In our domain, we modeled museum visitors as actors. As part of the DDD services within this bounded context, was a device gateway that held a web socket channel to visitor tablets. This proved very efficient and easy to reason about as domain entities within our system. In a previous post I described how we used the Concurrent.broadcast mechanism in Play to create a web socket connection. In this post, I will dive a little deeper and show you how you can model your actor and device gateway service to allow for easier unit testing of the web socket channel offered through Play’s Concurrent.broadcast.

To do this, we will use the following strategy:

    – Create a trait that handles creation and usage of Concurrent.broadcast channels

    – Using Traits, dependency inject this trait onto our actor representing the visitor entity

    – Create a controller with a method that will instantiate our actor and return back a broadcast channel

First, let’s create our trait whose single responsibility is the creation of the Concurrent.broadcast channels and usage of them:

Next, let’s create our actor worker to represent our visitor entity and inject our WebSocketChannel trait:

Next, let’s create a simple controller that will allow us to create a web socket connection:

Cool, with these pieces we can establish a web socket connection. Based on the logic in our RegisterSocket event handler within our actor, we should expect a JSON message with the keys ‘type’ and ‘data’, and their respective values. So how do we unit test this?

Well first off, let’s determine just what we need to test. Play already has tests to verify that it’s Conncurrent.broadcast mechanism works, so we don’t need to reinvent the wheel here. However, we do want to ensure the side effects of our RegisterSocket, which is that a message is pushed up the channel.

With the use of traits and DI, this becomes surprisingly easy!

First let’s create another trait that extends our WebSocketChannel trait that allows getting the message being sent:

Here we override the push method, keeping the argument and return types the same. The only difference is that we utilize a List data structure to model a channel, which itself is Queue (First in, First Out). With this trait we can write a simple test that verifies the side effect of our RegisterSocket event.

And that’s it! We have successfully tested the side effect of our actor’s web socket channel.

 

Could MTA Ever Make A Leap?

by

This past March, Leap launched a new “lounge-like” bus service in San Francisco. The Verge found it reminiscent of the reception area for a large tech company. I was reminded of WeWork, in their own words:

At Control Group, we reinvent services– public and private– by building capacity within the organizations that deliver them; we connect our efforts to a business need or revenue stream to ensure the changes are sustainable. Part of this process involves putting aside the constraints of an existing organization– thinking big – and then (sometimes lightly) reapplying reality until things fit.

So a thought experiment: consider the New York MTA, which serves one of the most dense cities in the United States, New York City. Could Leap become a model for MTA’s bus service, or is Leap forever destined to be a startup?

An overview of some key issues from our perspective:

Funding. Funding for “amenities” in the public sector is scarce. How can new revenues, e.g. from advertising, be earmarked towards funding amenities like Leap instead of being spent to maintain the status quo?

Mandate. A “customer-first” mandate needs to come from the top. Customer communications and marketing is an easy place to start; getting customer communications and operations together to affect service is a bigger challenge. Great things are happening at Sound Transit, LA Metro and Motivate.

Title VI. Requires equal service be delivered under Federally-funded programs. Leap doesn’t serve everyone. It isn’t handicap accessible. It doesn’t stop in all areas of the city. Could MTA leverage technology to help?

Cultural Plurality. As Leap imagines bringing a “living room” to a formerly sterile public space, can we all agree what a living room should look like? Design-led research and community inclusion can help build consensus around shared values.

Partnership. One solution to some of these issues is partnership with the private- and third- sectors. Partnership can externalize risk, helping government “take a chance”. Finding the right areas and structure for collaboration and partnership is critical.

Technology. Technology is absolutely essential to making a model like Leap work on a larger scale– especially as it might expand the reach of such a system via dynamic routing, dispatch and pickup requests. Imagine a bus that works like Uber.

Density/Connections. Currently, Leap only travels between two neighborhoods in San Francisco, making only 3-4 stops to pick-up passengers, depending on time of day. Dense residential and commercial areas, bikeshare and other “last mile” mobility solutions make limited stop services like Leap work.

The biggest barrier for Leap as a model for public transit is that Leap solves a different problem. Public transit has to serve everyone, Leap serves a very select few. Finding the aspects that are transferrable and those that are not will be the task of government and startups that provide transportation services (like Leap, Bridj, Uber and others) in the coming years.

At Control Group, we are defining the “responsive city”– a place where connectivity, interactive digital touchpoints and sensors might make a city more efficient and responsive to citizen needs. Self-driving cars aside, these technologies stand to fundamentally change the economics and operational models for public transit.

EMK Institute: Commit History

by

For the last two years, dozens of people at Control Group have been building the technology stack and implementation behind the recently opened Edward M. Kennedy Institute. Working with our design partner ESI Design, we created something that has never existed before: an interactive, immersive digital museum experience that includes a complex, tablet-driven role playing recreation of a Senate session in addition to multiple games and galleries in the chamber surround.

The 42-minute video above, synched to some of Ted Kennedy’s most iconic speeches (downloaded from http://tedkennedy.org/ownwords), is generated using gource off of our git logs. For the non-coders, what that means is every time we make a commit (ie: save a file) to one of our source-code repositories, the action appears on this visual map of our system.  The dots are files, the lines represent folders, and the people icons are (you guessed it) people.

Software is built by people, for people– many people sometimes, working together, crafting and refining and perfecting. You can somewhat see into our process in this short film. This movie focuses on 5 different repositories that cover a lot of what we did for the museum (but not all by any means, we have over a dozen repos for the project). The source code repositories– and where they are visualized in the movie– are detailed below:

Platform: (20 seconds) Primarily Scala – used to control and manage all museum interactions, API for internal and external data feeds. Like the project itself, this repo starts a long time ago and takes a while to get going, but once it does it really flies.

DevOps: (10 min) Build process and deployment, crucial for a project of this complexity. Also amazing to see how this was refined and improved over the last few months before launch.

Exhibits: (16 min) Everything up on the walls of the museum with a few exceptions. You’ll see a lot of activity in this one!

Webapp:  (25 min) Tablet web-based technology – AngularJS, HTML5, CSS, etc. This was also very active throughout the project, with many contributors.

Tablet: (33min) Android Application that hosts Webapp technology.

Everyone at Control Group who worked on this project is extremely proud of their contributions, and while there is no greater tribute than the physical institute itself, this movie shows some of the hard and constant work required to engineer something beautiful. Thanks to EMKI, ESI, Gigantic Mechanic and many others for the opportunity to work on this project and bring it alive.

EMK Institute: A Software Powered Senate

by

We developed the custom software to power the Edward M. Kennedy Institute, a highly interactive museum dedicated to the United States Senate. This software stack had to accommodate 22 unique interactive exhibits and 2 unique experiences. The experience we will be touching upon in this post is the Senate Immersion Module.

The Senate Immersion Module, or SIM, is a fully interactive simulation of a U.S. Senate voting session. Each participant has a tablet with which to cast votes on several potential bills, including the environment, immigration, and firearms, as well as nominees for federal posts.

One of the core parts of this experience is the ability to become a senator and work your way through the senate process to ultimately pass a bill. To start this, a player must be assigned a senator profile, which is sourced from an internal CMS that we built. By putting the content in a CMS, new content for the SIM can be created by the EMK staff and external partners over time. The distribution described below makes it easier for them to write a SIM that has a balanced mix of competing positions. As with any piece of software, there are certain requirements that must be met. In this case, a subset of the requirements is as follows:

  • Allow a maximum of 100 players
  • Allow a minimum of 24 players
  • Distribute players into 3 possible committees, 6 possible subcommittees, and 18 possible issue conferences.
  • The distribution of players must remain balanced along party lines for committees, subcommittees, and issue conferences.

To help illustrate, consider 100 players divided evenly between the 3 committees.

Screen Shot 2015-03-30 at 9.54.44 AM

This would give us 34 players in Committee A, 33 players in Committee B, and 33 players in Committee C.

Now, since each committee has 2 subcommittees, we have to evenly divide players into 6 subcommittees.

SIM Distribution (1)

And furthermore, each committee has 6 Issue Conferences in which players must be also be evenly distributed. (No diagram of that, but we would have 18 issue conferences with about 5 or 6 players each).

So it seems we have nested distributions where we must maintain balance along party lines and player counts for each respective distribution. Furthermore, the example above only shows for the base case of 100 players, but we know the number of players can range from 24 to 100. How should we go about this? Where do we even start?

As with any problem, the best first step is to break the problem down into smaller subproblems. Since it seems the committee distributions rely on the results of the subcommittee distributions and the subcommittee distribution relies on the results of the issue conference distributions, it seems the best place to start is with the issue conferences as it is the base sub-problem.

Players(Committee) = Players(SubCommittee1) + Players(SubCommittee2)

Players(SubCommitteeX) = Players(IssueConference1) + Players(IssueConference2) + Players(IssueConference3) + Players(IssueConference4) + Players(IssueConference5) + Players(IssueConference6)

So now we know where to start, but how do we start? Well, let’s consider which CS algorithms address our problem statement. There are 2 basic algorithms that come to mind:

Lets take a look at these algorithms, starting with the Partition problem. In the most basic sense, this algorithm returns whether or not a set of items can be partitioned into 2 or more equal sets. In the example given on wikipedia, they break the following set of numbers:

S = {3,1,1,2,2,1}

into two subsets that both equal 5:

S1 = {1,1,1,2} and S2 = {2,3}

Great! This seems similar to what we want to do, except with a set of Senators.

What about the Bin-Packing problem? This algorithm attempts to pack a set of items N into a set of Bins B. In the best case, all the items can be packed into a single bin. This is referred to as the Best-Fit. In the worst case, all the items are packed into all the available bins, which is referred to as Worst-Fit. So if for example we had three 3-gallon buckets, and one 3 gallon jug of water, our cases would be:

  • Best-Fit: Pour all 3 gallons into the first bucket
  • Worst-Fit: Pour 1 gallon into each of the 3 buckets.

So both these algorithms seem the same, which one do we go with? Well remember, the Partition problem aims to tell you IF equal partitions are possible, while the Bin-Packing aims to give you the ways you can place a set of items into a certain number of bins. So it seems Bin-Packing is the algorithm for us! Now, we need to decide which case of bin packing we need, Best or Worst? Since Worst-Fit attempts to utilize all bins, this means a more uniform distribution for our Issue Conferences, so Worst-Fit it is!

Now taking our base sub problem of issue conference assignment, we must iterate through each issue conference bin and place a senator into that bin, while maintaining party balance per bin. Now we could come up with a complex algorithm that does the packing while also having logic to maintain balance, but part of the core ideas in computer science, is the idea of single responsibility. This means we will have our bin packing function only concerned about placing senators into issue conference bins. We will place the logic for maintaining balance inside another function. To simplify things a little, we will maintain balance as best as possible by packing each party one at a time. So if our two parties are Democrats and Republicans, we will fit 1 Democrat per issue conference first, then 1 Republican per issue conference, then repeat until we have fit all our senators. Below is the function written in Scala that does the Bin packing.

[1] Here we define a function called ‘fitSenatorBasedOnRarity’. The rarity has to deal with the frequency of issues conferences to senators who care about that issue, with the order being the most rare issue conference to the least rare issue conference. This is calculated early on in another function, with the result being passed in as a variable utilizing a Map data structure of Committees to Issue Conferences, called committeesWithIssueConfs.  The second argument to the function is a Set of senators, called senators. The final argument is a String indicating the party we are currently dealing with within the algorithm, called party.

[2] Scala is a functional language (http://en.wikipedia.org/wiki/Functional_programming) whose default types are immutabble. However because we want to remove senators that are placed into an issue conference, we convert from an immutable Set to a mutable Set.

[3] Here we take our Map data structure and do two iterations, the first over the keys of the map, which are the Committees, and the second over the issue conferences for that respective Committee. It may help to think of this as a loop within a loop.

[4] At this step we use our mutable Set of senators and loop through them to find a senator whose party/faction equals the party we are currently working on and also who cares about this particular issue conference via the function ‘doesMatchPositionRule’.

[5] At the next step we check the results of step 4 to see if we found a Senator that matches the criteria or if we did not.

[6] In the case we have found some senator that matches the criteria for step 4, so we add them to the issue conference for the current committee.

[7] We then  remove them from the Set of available senators so they are not used again in the next loop iteration.

[8] At this step we have completed our iterations and have removed quite a few senators. We return back a now reduced Set of senators. This reduced Set can be sent back into the ‘fitSenatorBasedOnRarity’ function for another pass with a different party.

So we keep utilizing this function by passing in the updated Set of senators and alternating between the parties until we have no more senators left. Now that we have our issue conferences, we have solved our first sub problem and can use that result to solve the next sub problem, Bin Packing for Sub Committees. Below is the function for Bin Packing the Sub Committees.

[1] Here we define a function named ‘defineSenatorTrack’. Although this funciton is doing Bin Packing for our Sub Committees, we got clever and also used it to Bin Pack for our Committees as well, which in turn completes our senator track for a particular senator.

[2] The variable committeeIssueSenatorMap is the results of step 6 of the previous function. It is also based on the Map data structure, but is a Map of Committee to Issue Conferences, where each Issue Conference is mapped to Set of senators. Here we iterate over each Committee and iterate again over each issue Conference in that respective Committee, then iterate again over each Senator in that respective Issue Conference.

[3] Here we utilize the functional concept of folding to calculate the sum of all the current senators in all Issue Conferences within the respective Committee of a particular party, then divide that number by 2 to determine the maximum number of senators for a particular party that can be in each Sub Committee.

[4] At this step we utilize another function called ‘doesSenatorFitInSubCommittee’ which checks if the result of adding this particular senator to Sub Committee 1 keeps the allotted number of factions per committee under the maximum calculated in step 3. The details of this method are not important, just know that it uses some functional programming concepts to quickly calculate this and return either true or false, indicating if the senator can be placed into a certain Sub Committee.

[5] If the results of step 4 are true, meaning that the senator can be placed into Sub Committee 1, then we update another variable based on the Map data structure called ‘senatorTrackMap’, and place this senator within Sub Committee 1 as well as place them within the current Committee.

[6] This step is the same as step 4, but instead of calling ‘doesSenatorFitInSubCommittee’ with Sub Committee 1, we call the function with Sub Committee 2 to determine if the senator can be placed within Sub Committee 2 without breaking party balance.

[7] If the results of step 6 are true, then we can update the ‘senatorTrackMap’ variable to place this current senator within Sub Committee 2, as well as placing them within the current Committee.

[8] If the results of step 4 and step 6 are both false, then we log that we cannot place this senator. There is additional logic not shown here to deal with this case.

At this point we have successfully completed the second sub problem and cleverly completed the first problem of Bin Packing into the Committees as well. As you can see the use of the Bin Packing algorithm within our software is somewhat different that the generalized form shown in Wikipedia and in text books. Instead of packing numbers into Bins, we packed Objects that encapsulated information about a Senator. Also instead of just using the numbers to determine the weights of each item and the size of a Bin, we used meta data, such as the Senator’s Party to derive a numerical weight and the total number of players to drive the maximum size of each Bin.

It was pretty cool to use a fundamental algorithm such as Bin Packing to describe how and why we placed Senators into Committees, Sub Committees, and Issue Conferences for our Senate Immersion Module software. I hope that this post shows how applicable these algorithms are outside of their generalized form. Data structures and Algorithms are one of the key, if not the most important aspects of Computer Science and in turn Software Engineering. Having knowledge of these can help you better figure out how to approach and solve both simple and complex problems. I would sincerely recommend everyone to take some time out to learn some of these basic algorithms. You may not fully understand them the first time around, but take it slowly and eventually you will develop not only an understanding but an appreciation for them as they are at the heart of every piece of software you use daily.

Helpful Resources:

Edward M. Kennedy Institute: For the people, by the people

by

EMK InstituteToday, the Edward M. Kennedy Institute for the United States Senate (EMK Institute) opened its doors to educate students, invigorate public discourse and inspire visitors to become more active in their own communities, through a bold concept of accessibility to the Senate experience. The Institute’s concept, design, and production are the result of a remarkable collaboration between Senator Kennedy, his wife Victoria Reggie Kennedy, Ed Schlossberg and a consortium of leading technology, design, gaming, education and media firms, of which Control Group is extremely proud to be a part of.

Senator Kennedy was a hero for the country, and a staunch defender of the democratic process, and fulfilling his vision for an educational institution dedicated to that process is a dream come true. To be defining the next iteration of the museum and institution in the 21st Century, as a connected, intelligent, public experience, is also a proud moment for all the people at Control Group that worked so hard to get us here.

Public space is becoming the new frontier of technology touchpoints. Last year we installed digital information kiosks in the New York City subway system. The year before we built a mobile engagement tool to capture public opinion for the mayoral transition in NYC. And this year we’ll begin to roll out LinkNYC, the largest free municipal wi-fi and open platform in the world. All of these initiatives, including the EMK Institute, are about creating digital infrastructure to extend placemaking and engagement in cities, regardless of physical location.

Throughout history, to experience museums, libraries and government buildings meant a trip to a specific space. An experience not historically accessible to all, either because of travel, cost, privacy and security concerns. The Internet and related technologies have increased the potential reach and accessibility by providing content and an experience previously only found within their walls. As new technologies, new tools, and new approaches emerge, the reach and impact of these institutions will become even greater– and people will be able to actively participate and experience them, whether within their walls, or in a grade-school or university classroom. This extensibility was core to the vision for EMK Institute.

While EMK Institute is a beautiful physical building designed by Rafael Viñoly and located in Boston, the concept of the Institute was built with digital technology at its core because that is the foundation of collaboration and connectedness in today’s world – and fostering collaboration and connectedness is the very heart of the mission of the Institute. So we approached the technical design and development of the museum as if it were digital space, rather than a physical space with digital exhibits. This allows the Institute to be more than its four walls and website. It’s a platform that allows visitors to contribute to it, and participate in it, where they become are the art, theater, and exhibition.

Inside the museum, this means that each visitor is part of a group and individual experience. Mobile tablets provide access to personalized content and act as a game controller, so visitors can make sound arguments and navigate negotiations with their peers. It also allows them to vote, while keeping track of all of the interactions and accomplishments. Digital exhibits and displays show each visitor’s contribution to the collective process and provides a record of their accomplishments.

For audiences and participants “outside” of the museum, we built the software on an open framework and created a toolkit that allows both EMK Institute and external organizations to create new content and simulations to plug into the system. This ensures that content stays fresh and new insights and perspectives are included. The toolkit will also enable schools from around the country to conduct versions of the simulations in their own classrooms. The openness and accessibility of the museum reflects the very ethos of Institute– that collaboration is necessary to achieve the greater good, inside and outside of the Senate walls.

We give a huge thanks to the key partners in this project. This all came together through intense collaboration with the EMK Institute, Ed Schlossberg and ESI for vision and exhibit design, Richard Lewis Media Group with exhibit content strategy and production, and Gigantic Mechanic, who created the  game mechanics of the group activities and simulations. Of course, a deep thanks goes to our entire CG team who worked on this, including product managers, strategists, full-stack and front-end developers, mobile developers, interaction designers, design technologists, DevOps, network and server engineers, project managers, QA, and more.

As a company, we feel a deep spiritual kinship with Senator Kennedy as a doer, maker, and innovator, not shying away from tough problems, and not content with the status quo. Being a company who is actively engaged in pushing the edges or building the future is not easy. Being a politician who is intent on having fair, public conversation about important and difficult issues, finding a middle ground and passing legislation is not an easy or popular road either. We are humbled by and laud Senator Kennedy’s efforts, and others in government present and past, who are driving change with that long term view, creating lasting value for us all as a progressive society. Innovators see the world in the same way.

Congratulations to Senator Edward Kennedy’s family, to the EMK Institute and staff, to the state of Massachusetts, to Senators past and present, and to the teams who worked so hard to make this dream come true.