Kafka Partition

 In previous examples, the ProducerRecord objects we created included a topic name,

key, and value. Kafka messages are key-value pairs and while it is possible to create a

ProducerRecord with just a topic and a value, with the key set to null by default,

most applications produce records with keys. Keys serve two goals: they are addi‐

tional information that gets stored with the message, and they are also used to decide

which one of the topic partitions the message will be written to. All messages with the

same key will go to the same partition. This means that if a process is reading only a

subset of the partitions in a topic (more on that in Chapter 4), all the records for a

single key will be read by the same process. To create a key-value record, you simply

create a ProducerRecord as follows:

ProducerRecord<Integer, String> record =

 new ProducerRecord<>("CustomerCountry", "Laboratory Equipment", "USA");

When creating messages with a null key, you can simply leave the key out:

ProducerRecord<Integer, String> record =

 new ProducerRecord<>("CustomerCountry", "USA");

Here, the key will simply be set to null, which may indicate that a customer

name was missing on a form.

When the key is null and the default partitioner is used, the record will be sent to

one of the available partitions of the topic at random. A round-robin algorithm will

be used to balance the messages among the partitions.

If a key exists and the default partitioner is used, Kafka will hash the key (using its

own hash algorithm, so hash values will not change when Java is upgraded), and use

the result to map the message to a specific partition. Since it is important that a key is

always mapped to the same partition, we use all the partitions in the topic to calculate

the mapping—not just the available partitions. This means that if a specific partition

is unavailable when you write data to it, you might get an error. This is fairly rare, as

you will see in Chapter 6 when we discuss Kafka’s replication and availability.

The mapping of keys to partitions is consistent only as long as the number of parti‐

tions in a topic does not change. So as long as the number of partitions is constant,

you can be sure that, for example, records regarding user 045189 will always get writ‐

ten to partition 34. This allows all kinds of optimization when reading data from par‐

titions. However, the moment you add new partitions to the topic, this is no longer

guaranteed—the old records will stay in partition 34 while new records will get writ‐

ten to a different partition. When partitioning keys is important, the easiest solution

is to create topics with sufficient partitions (Chapter 2 includes suggestions for how

to determine a good number of partitions) and never add partitions.

Kafka Producer

We start producing messages to Kafka by creating a ProducerRecord, which must

include the topic we want to send the record to and a value. Optionally, we can also

specify a key and/or a partition. Once we send the ProducerRecord, the first thing the

producer will do is serialize the key and value objects to ByteArrays so they can be

sent over the network.

Next, the data is sent to a partitioner. If we specified a partition in the

ProducerRecord, the partitioner doesn’t do anything and simply returns the partition

we specified. If we didn’t, the partitioner will choose a partition for us, usually based

on the ProducerRecord key. Once a partition is selected, the producer knows which

topic and partition the record will go to. It then adds the record to a batch of records

that will also be sent to the same topic and partition. A separate thread is responsible

for sending those batches of records to the appropriate Kafka brokers.

When the broker receives the messages, it sends back a response. If the messages

were successfully written to Kafka, it will return a RecordMetadata object with the

topic, partition, and the offset of the record within the partition. If the broker failed

to write the messages, it will return an error. When the producer receives an error, it

may retry sending the message a few more times before giving up and returning an

error

Once we instantiate a producer, it is time to start sending messages. There are three

primary methods of sending messages:

Fire-and-forget

We send a message to the server and don’t really care if it arrives succesfully or

not. Most of the time, it will arrive successfully, since Kafka is highly available

and the producer will retry sending messages automatically. However, some mes‐

sages will get lost using this method.

Synchronous send

We send a message, the send() method returns a Future object, and we use get()

to wait on the future and see if the send() was successful or not.

Asynchronous send

We call the send() method with a callback function, which gets triggered when it

receives a response from the Kafka broker. 

acks

The acks parameter controls how many partition replicas must receive the record

before the producer can consider the write successful. This option has a significant

impact on how likely messages are to be lost. There are three allowed values for the

acks parameter: 0, 1, all

Update MySQL database engine

 SELECT ENGINE FROM information_schema.TABLES WHERE TABLE_NAME = 'tabel_name' AND TABLE_SCHEMA='db_name';

ALTER TABLE table_name ENGINE = INNODB;

install java in ubuntu

 #Login as root

sudo su

#create jdk directory
mkdir /opt/jdk

#uncompress, change to your file name
tar -zxf jdk-8u5-linux-x64.tar.gz -C /opt/jdk

#check if files are there
#ls /opt/jdk

#update alternatives so the command java point to the new jdk 
update-alternatives --install /usr/bin/java java /opt/jdk/jdk1.8.0_05/bin/java 100


#update alternatives so the command javac point to the new jdk 
update-alternatives --install /usr/bin/javac javac /opt/jdk/jdk1.8.0_05/bin/javac 100

#check if java command is pointing to " link currently points to /opt/jdk/jdk1.8.0_05/bin/java"
update-alternatives --display java

#check if java command is pointing to " link currently points to /opt/jdk/jdk1.8.0_05/bin/javac"
update-alternatives --display javac

#check if java is running
java -version

sudo  update-alternatives --config java

Docker sample

 FROM maven:3.6.0-jdk-8-alpine AS MAVEN_BUILD

MAINTAINER Sushil
COPY pom.xml /build/
COPY src /build/src/
WORKDIR /build/
RUN mvn package
FROM openjdk:8-jre-alpine
WORKDIR /app
COPY --from=MAVEN_BUILD /build/target/name.jar /app/
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "name.jar"]

docker command

 

sudo docker image build -t sushil/verification ./

sudo docker container run -d -p 8087:8080 sushil/verification

list of images::

sudo docker ps

sudo docker logs -f d44b2b216e16

sudo docker exec -it d44b2b216e16 sh

tail -f verification.log

help:  https://dzone.com/articles/build-package-and-run-spring-boot-apps-with-docker

stop::

docker stop <container-name>

Cassandra

 https://www.simplilearn.com/tutorials/big-data-tutorial/cassandra-architecture