DataDotz BigData Weekly

DataDotz BigData Weekly

Confluent released first preview version of Confluent Platform
This first preview release introduces powerful new capabilities for KSQL (streaming SQL for Apache Kafka®) and Confluent Control Center. Confluent Control Center provides features such as UI for KSQL, Broker Configuration, Topic Inspection, Consumer Lag.  They have also made several improvements to KSQL REST API. Additional KSQL features such as flexible timestamp handling, non-windowed aggregate functions(SUM, COUNT) on table. Preview release also includes protection on both tables and streams.
https://www.confluent.io/blog/introducing-confluent-platform-preview-releases/
 
Continue reading

Read More
DataDotz BigData Weekly

DataDotz Bigdata Weekly

Apache Spark
=====

we start to dive into the details of running custom versions of Spark, it’s important to note if all you need to do is run a “supported” version of Spark on Google Cloud Dataproc or Spark on Kubernetes there are much easier options and guides out there for you. Continue reading

Read More
DataDotz BigData Weekly

DataDotz Bigdata Weekly

Apache Spark
=====

Structured Streaming in Apache Spark 2.0, it has supported joins (inner join and some type of outer joins) between a streaming and a static DataFrame/Dataset. With the release of Apache Spark 2.3.0, now available in Databricks Runtime 4.0 as part of Databricks Unified Analytics Platform, we now support stream-stream joins. Continue reading

Read More