Speakers | |
---|---|
Nils Grunwald | |
Schedule | |
Day | Sunday |
Room | AW1.125 |
Capacity | 76 |
Start time | 10:20 |
End time | 11:05 |
Duration | 00:45 |
Info | |
Track | Graph Processing Devroom |
Using Cascalog and Hadoop for rapid graph processing and exploration
Graphs are becoming increasingly popular as ways of modeling a wide variety of systems. As such, the label "graph processing" also covers a range of objectives and architectural constraints. At [Linkfluence][http://us.linkfluence.net/], we use graph processing on datasets produced with very different systems (Web crawler, Twitter and Facebook API, feed aggregator, etc.) We spend a lot of time doing exploratory programming, trying to use our eclectic datasets in interesting ways, and processing our data in asynchronous workflows.
We have come to see [Hadoop][http://hadoop.apache.org/] and the processing framework [Cascalog][https://github.com/nathanmarz/cascalog] as essential tools in our toolbox when dealing with graphs, since it gives us architectural flexibility, scalability and the possibility of rapid prototyping.
Cascalog is an open source framework built on top of Hadoop and [Cascading][http://www.cascading.org/]. Its syntactic and semantic roots come from Datalog and Prolog, which have been succesfully applied for a long time in the manipulation of graphs. Also, its ability to directly embed the expressive [Clojure][http://clojure.org/] language allows to very easily define custom operations and ad-hoc processing.
In this talk, we will present the framework, consider its advantages and drawbacks when compared to other approaches, show concrete exemples of usage for graph processing and how we use them to complement graph databases.
Concurrent events:
Next (up to 3) talks in the same room (AW1.125):
Events that start after this one (within 30 minutes):
When | Event | Track | Where |
---|---|---|---|
11:05-11:45 | Perlito | Perl | AW1.121 |
11:10-11:55 | Birds of a feather - Graph processing, future trends! | Graph Processing | AW1.125 |
11:10-11:55 | Introduction to HelenOS | Microkernel OS | K.3.201 |
11:15-11:55 | From zero to VoIP provider in 15 minutes | Telephony and Communications | H.2213 |
11:20-11:35 | Amarok | CrossDesktop | H.1308 |
11:20-11:35 | LibrePlan: Open Web Planning | Lightning Talks | Ferrer |
11:30-11:55 | oVirt Engine Core: Internals and Infrastructure | Virtualization and Cloud | Chavanne |
11:30-11:55 | Build simple and complex replication clusters with Tungsten Replicator | MySQL and Friends | H.1309 |
11:30-12:00 | Can I legally do that? | Free Java | K.4.401 |
11:30-12:00 | Wayland Q & A for toolkit developers. | X.org+OpenICC | K.3.401 |
11:30-12:00 | Introducing the Metrics Data Ping | Mozilla | UD2.218A |
11:30-12:30 | Working with contributor communities (round table) | CrossDistribution | H.1301 |