big data – Seite 2 – Zukunft-Innovation-Technik (ZuInnoTe)

Ethereum & Analytics: Explore the blockchain using Hadoop, Hive, Flink and Spark

Nov 14, 2017

—

von

HadoopCryptoLedger release 1.1.0 added support for another well-known cryptocurrency: Ethereum and its Altcoins. Of course similar to its Bitcoin & Altcoin support you can use the library with many different frameworks in the Hadoop ecosystem: Hadoop MR Apache Hive Apache Flink Apache Spark and Apache Spark Datasource API Furthermore, you can use it with various…

Mapred vs MapReduce – The API question of Hadoop and impact on the Ecosystem

Okt 20, 2017

—

von

Jörn Franke

in analytics, big data, flink, graph, hive, office, spark, tech

I will describe in this blog post the difference between the mapred.* and mapreduce.* API in Hadoop with respect to the custom InputFormats and OutputFormats. Additionally I will write on the impact of having both APIs on the Hadoop Ecosystem and related Big Data platforms, such as Apache Flink, Apache Hive and Apache Spark. Finally,…

Big Data Analytics on Bitcoin‘s first Altcoin: NameCoin

Okt 10, 2017

—

von

Jörn Franke

in altcoin, analytics, big data, bitcoin, flink, hive, tech

This blog post is about analyzing the Namecoin Blockchain using different Big Data technologies based on the HadoopCryptoLedger library. Currently, this library enables you to analyze the Bitcoin blockchain and Altcoins based on Bitcoin (incl. segregated witness), such as Namecoin, Litecoin, Zcash etc., on Big Data platforms, such as Hadoop, Hive, Flink and Spark. A…

Templates, low footprint mode, improved integration with Spark for the HadoopOffice library for reading/writing Excel files on Big data platforms

Aug 1, 2017

—

von

Jörn Franke

in analytics, big data, office, tech

Although it seems to be that it was only a small improvement, version 1.0.4 of the HadoopOffice library has a lot of new features for reading/writing Excel files: Templates, so you can define complex documents with diagrams or other features in MSExcel and fill it with data or formulas from your Big Data platform in…

Leverage the Power of Apache Flink to analyze the Bitcoin Blockchain

Mai 24, 2017

—

von

Jörn Franke

in analytics, big data, bitcoin, flink, tech

The hadoopcryptoledger library has been enhanced with a datasource for Apache Flink. This means you can use the Big Data processing framework Apache Flink to analyze the Bitcoin Blockchain. It also includes an example that counts the total number of transactions in the Bitcoin blockchain. Of course given the power of Apache Flink you can think…

Reading/Writing Excel documents with the HadoopOffice library on Hadoop and Spark – First release

Jan 8, 2017

—

von

Jörn Franke

in analytics, big data, office, tech

Reading/Writing office documents, such as Excel, has been always challenging on Big data platforms. Although many libraries exist for reading/writing office documents, they have never been really integrated in Hadoop or Spark and thus lead to a lot of development efforts. There are several use cases for using office documents jointly with Big data technologies:…

Lambda, Kappa, Microservice and Enterprise Architecture for Big Data

Nov 11, 2016

—

von

Jörn Franke

in big data, business, emerging trends

A few years after the emergence of the Lambda-Architecture several new architectures for Big Data have emerged. I will present and illustrate their use case scenarios. These architectures describe IT architectures, but I will describe towards the end of this blog the corresponding Enterprise Architecture artefacts, which are sometimes referred to as Zeta architecture. Lambda…

Sneak Preview – HadoopOffice: Processing Office documents using the Hadoop Ecosystem – The example of Excel files

Okt 22, 2016

—

von

Jörn Franke

in analytics, big data, hive, office, tech

I present in this blog post the sneak preview of the hadoopoffice library that will enable you to process Office files, such as MS Excel, using the Hadoop Ecosystem including Hive/Spark. It currently contains only an ExcelInputFormat, which is based on Apache POI. Additionally, it contains an example that demonstrates how an Excel input file…

Spark+Scala+Graphx: Analyzing the Bitcoin Transaction Graph

Sep 20, 2016

—

von

Jörn Franke

in analytics, big data, bitcoin, graph, tech

The hadoopcryptoledger library provides now an example how you can generate a Bitcoin Transaction Graph using the Big Data graph analysis technologies Spark+Scala+Graphx. Basically it demonstrates how to read the Bitcoin Blockchain from HDFS, transform it into a graph with Bitcoin addresses as vertices and transactions between them as edges. The example returns the 5…

Hive & Bitcoin: Analytics on Blockchain data with SQL

Apr 28, 2016

—

von

Jörn Franke

in analytics, big data, bitcoin, data warehouse, hive, tech

You can now analyze the Bitcoin Blockchain using Hive and the hadoopcryptoledger library with the new HiveSerde plugin. Basically you can link any data that you loaded in Hive with Bitcoin Blockchain data. For example, you can link Blockchain data with important events in history to determine what causes Bitcoin exchange rates to increase or…

Kategorie: big data