Big Data

NIST Big Data WG Use Case Survey

This is the second version of a use case survey conducted by the NIST Big Data Working Group. If you are working on, or planning a Big Data project, we would like to learn more about your use case.

The survey can take as long as 90 minutes to complete, depending on the extend of the deployment, complexity of applications, infrastructure and security.


[iframe src=”” id=”nistusecase2″ name=”NISTUseCase2″ width=”760″ height=”500″ frameborder=”0″ scrolling=”no” marginheight=”0″ marginwidth=”0″]


Read More

NTIA Meeting on Best Practices for UAS Privacy, Transparency and Accountability

Full title: Notice of Multistakeholder Process to Develop Best Practices for Privacy, Transparency, and Accountability Regarding Commercial and Private Use of Unmanned Aircraft Systems. Following is from the NTIA announcement verbatim:

NTIA will convene meetings of a multistakeholder process concerning privacy, transparency, and accountability issues regarding commercial and private use of unmanned aircraft systems. The meetings will be held on August 3, 2015; September 24, 2015; October 21, 2015; and November 20, 2015 from 1 p.m. to 5:00 p.m., Eastern Time. The meetings will be held in the Boardroom at the American Institute of Architects, 1735 New York Avenue NW, Washington, DC 20006.

For further information please contact John Verdi, National Telecommunications and Information Administration, U.S. Department of Commerce, 1401 Constitution Avenue, NW, Room 4725, Washington, DC 20230; telephone (202) 482-8238; email [email protected]. Please direct media inquiries to NTIA’s Office of Public Affairs, (202) 482-7002; email [email protected].

Read More

Meetup: Spotify’s Recommendations Lambda Architecture

Meeting announcement from Meetup via the Organizer, Eugene Dvorkin:

At Spotify, one of the ways we surface personalized music recommendations to users is via the Discover page. The recommendations are powered by training large scale Matrix Factorization models on user listening history. Batch hadoop jobs run daily to build latent vectors for users, tracks, artists and albums.

However, this setup doesn’t incorporate intra-day listening history. Furthermore, new users that log into Spotify and stream music will not receive recommendations until their second day using the application. To solve the problem, we leverage Storm and process user listens in real-time; this allows us to surface recommendations to new users as soon as they start listening to songs on Spotify. We shall discuss our framework, how we use Storm and challenges we faced building this system.


Read More