Saturday, July 29, 2017

HackerRank Equals

Christy is interning at HackerRank. One day she has to distribute some chocolates to her colleagues. She is biased towards her friends and may have distributed the chocolates unequally. One of the program managers gets to know this and orders Christy to make sure everyone gets equal number of chocolates.
But to make things difficult for the intern, she is ordered to equalize the number of chocolates for every colleague in the following manner,
For every operation, she can choose one of her colleagues and can do one of the three things.
  1. She can give one chocolate to every colleague other than chosen one.
  2. She can give two chocolates to every colleague other than chosen one.
  3. She can give five chocolates to every colleague other than chosen one.
Calculate minimum number of such operations needed to ensure that every colleague has the same number of chocolates. 
Input Format
First line contains an integer  denoting the number of testcases.  testcases follow.
Each testcase has  lines. First line of each testcase contains an integer  denoting the number of colleagues. Second line contains N space separated integers denoting the current number of chocolates each colleague has.
Constraints


Number of initial chocolates each colleague has < 
Output Format
 lines, each containing the minimum number of operations needed to make sure all colleagues have the same number of chocolates.
Sample Input
1
4
2 2 3 7
Sample Output
2
Explanation
1st operation: Christy increases all elements by 1 except 3rd one
2 2 3 7 -> 3 3 3 8
2nd operation: Christy increases all element by 5 except last one
3 3 3 8 -> 8 8 8 8


       

import java.util.Scanner;

/**
 * Created by vdokku on 7/29/2017.
 */
public class Equal {

    static boolean DBG = false;

    // Min number in the Array.
    static int min;

    //Find the minimised action counts
    public static int MinRound(int[] counts) {

        int[][] results = new int[counts.length][3];
        for (int i = 0; i < counts.length; i++) {
            for (int j = 0; j < 3; j++) {
                int delta = counts[i] - min + j; // calculating the DELTA is important.
                results[i][j] = 0;
                while (true) {
                    // Greedy approach
                    if (delta >= 5) {
                        delta -= 5;
                        results[i][j]++;
                    } else if (delta >= 2) {
                        delta -= 2;
                        results[i][j]++;
                    } else if (delta >= 1) {
                        delta -= 1;
                        results[i][j]++;
                    } else {
                        break;
                    }
                }
            }
        }
        int finalResult = -1;
        // Compare results from different baseline cases (keep min, take 1, 2 ).
        for (int i = 0; i < 3; i++) {
            int subFinal = 0;
            for (int j = 0; j < counts.length; j++) {
                subFinal += results[j][i];
                if (DBG) System.out.format("results[%d][%d] = %d \n", j, i, results[j][i]);
            }
            if (DBG) System.out.println(subFinal);
            if (finalResult < 0 || finalResult > subFinal) {
                finalResult = subFinal;
            }
        }

        return finalResult;
    }

    public static void main(String[] args) {

        int casesCount = 0;

        Scanner s = new Scanner(System.in);

        if (s.hasNextInt()) {
            casesCount = s.nextInt();
        }
        s.nextLine(); // throw away the newline.

        int[] outputArrary = new int[casesCount];

        for (int i = 0; i < casesCount; i++) {
            int count = 0;
            if (s.hasNextInt()) {
                count = s.nextInt();
            }
            s.nextLine();

            int[] numbers = new int[count];
            min = -1;
            for (int j = 0; j < count; j++) {
                if (s.hasNextInt()) {
                    numbers[j] = s.nextInt();
                    // get min value from input array
                    if (numbers[j] < min || min < 0) {
                        min = numbers[j];
                    }
                } else {
                    System.out.println("You didn't provide enough numbers");
                    break;
                }
            }
            //SortCounts(numbers);
            outputArrary[i] = MinRound(numbers);

        }

        for (int i = 0; i < casesCount; i++) {
            System.out.println(outputArrary[i]);
        }
    }
}

       
 

Thursday, July 27, 2017

LEARN complex HIVE Queries

When working with the complex hive queries which involves different analytical functions.
Here is a sample query.

       

SELECT
mag.co_magasin,
dem.id_produit                                  as id_produit_orig,
pnvente.dt_debut_commercial                     as dt_debut_commercial,
COALESCE(pnvente.id_produit,dem.id_produit)     as id_produit,
min(
CASE WHEN dem.co_validation IS NULL THEN 0 ELSE 1 END
)                                               as flg_demarque_valide,
sum(CASE WHEN dem.co_validation IS NULL THEN 0 ELSE cast(dem.mt_revient_ope AS INT) END)
           as me_dem_con_prx_cs,
0                                               as me_dem_inc_prx_cs,
0                                               as me_dem_prov_stk_cs,
sum(CASE WHEN dem.co_validation IS NULL THEN 0 ELSE cast(dem.qt_demarque AS INT) END)
           as qt_dem_con,
0                                               as qt_dem_inc,
0                                               as qt_dem_prov_stk,
RANK() OVER (PARTITION BY mag.co_magasin, dem.id_produit ORDER BY pnvente.dt_debut_commercial DESC, COALESCE(pnvente.id_produit,dem.id_produit) DESC) as rang
from default.calendrier cal
INNER JOIN default.demarque_mag_jour dem
ON  CASE WHEN dem.co_societe = 1 THEN 1 ELSE 2 END = '${hiveconf:in_co_societe}'
AND dem.dt_jour    = cal.dt_jour
LEFT OUTER JOIN default.produit_norm pn
ON  pn.co_societe = dem.co_societe
AND pn.id_produit = dem.id_produit
LEFT OUTER JOIN default.produit_norm pnvente
ON  pnvente.co_societe = pn.co_societe
AND pnvente.co_produit_rfu = pn.co_produit_lip
AND pnvente.co_type_motif='05'
INNER JOIN default.kpi_magasin mag
ON  mag.co_societe = '${hiveconf:in_co_societe}'
AND mag.id_magasin = dem.id_magasin
WHERE cal.dt_jour = '${hiveconf:in_dt_jour}'
AND NOT (dem.co_validation IS NULL AND cal.dt_jour > from_unixtime(unix_timestamp()-3*60*60*24, 'ddmmyyyy'))
-- JYP 4.4
AND dem.co_operation_magasin IN ('13','14','32')
GROUP BY
mag.co_magasin,
dem.id_produit,
pnvente.dt_debut_commercial,
COALESCE(pnvente.id_produit,dem.id_produit)

       
 










Tuesday, July 18, 2017

com.google.common.collect.Lists provide different methods.

com.google.common.collect;
Lists


public static <T> List<T> reverse(List<T> list) {
    if (list instanceof ImmutableList) {
      return ((ImmutableList<T>) list).reverse();
    } else if (list instanceof ReverseList) {
      return ((ReverseList<T>) list).getForwardList();
    } else if (list instanceof RandomAccess) {
      return new RandomAccessReverseList<T>(list);
    } else {
      return new ReverseList<T>(list);
    }
  }


 @Override public List<T> subList(int fromIndex, int toIndex) {
      checkPositionIndexes(fromIndex, toIndex, size());
      return reverse(forwardList.subList(
          reversePosition(toIndex), reversePosition(fromIndex)));
    }

public static <T> List<List<T>> partition(List<T> list, int size) {
    checkNotNull(list);
    checkArgument(size > 0);
    return (list instanceof RandomAccess)
        ? new RandomAccessPartition<T>(list, size)
        : new Partition<T>(list, size);
  }




why variables inside foreach loop of java8 should be final?

The Java Memory Model has very important property: it guarantees that local variables and method parameters are never writable by another thread. This adds much safety to multi-threading programming. However when you create a lambda (or an anonymous class), nobody knows how it will be used. It can be passed to another thread for execution (for example, if you use parallelStream().forEach(...)). Were it possible to modify the local variable that important property would be violated. Not the thing the Java language developers would sacrifice.

Usually when you are using lambdas, you are trying to program in functional way. But in functional programming mutable variables are considered bad practice: it's better to assign every variable only once. So trying to modify the local variable actually smells. Use various stream reduction methods instead of forEach to produce a good functional code.

https://stackoverflow.com/questions/16635398/java-8-iterable-foreach-vs-foreach-loop
https://stackoverflow.com/questions/31801313/why-variables-inside-foreach-loop-of-java8-should-be-final



Difference between == and === in scala,spark


The "==" is using the equals methods which checks if the two references point to the same object. The definition of "===" depends on the context/object. For Spark , "===" is using the equalTo method.

== returns a boolean
=== returns a column (which contains the result of the comparisons of the elements of two columns)

Quick Sample: Normally SchemaRDD is as RDD[org.apache.spark.sql.Row] , extracting the information out of schemaRDD is by using rec.getString(column number).


.filter(rec => (rec.getString(5) == "" || rec.getString(5) == null)).map(rec => (rec.getString(1))).

Mergin the PART file to a single file. You can use the same option of .repartition(1) or coalesce(1) to save only 1 part file.

hadoop fs -getmerge "HDFS_LOCATION_PART_FILES_LIST"  "NFS_LOCATION_SINGLE_PART_FILE"

Tuesday, July 11, 2017

Hadoop Certification Plan

Hive AVRO
Hive Parquet

Hive Window Operation.
Hive GroupBy

Removing the header of any (TEXT/CSV) files can be done in two ways.

stocksData.take(12).foreach(println)


Stick with the First approach.

1)  mapPartitionsWithIndex
 Ex: stocksData.mapPartitionsWithIndex{(index, itr) => if (index ==0) itr.drop(1) else itr}.take(12).foreach(println)



2)  mapPartitions
 Ex: stocksData.mapPartitions(itr => itr.drop(1)).take(12).foreach(println)



Simple commands that needs to be remembered.
  1. import sqlContext.implicits._
  2. spark-shell --packages com.databricks:spark-avro_2.10:2.0.1
  3. import com.databricks.spark.avro._
  4. import org.apache.spark.sql.functions._

Different ways of reading a Text file from HDFS.
Considering the input source files are comma separated.

       

val ordersDF = sc.textFile("/PATH").map(_.split(',')).map(p => orders(p(0).toInt,p(1),p(2).toInt,p(3))).toDF()
val rdd1 = sc.textFile("/Path").map(rec => {  val rec1 =  rec.split(',')  orders(rec1(0).toInt,rec1(1),rec1(2).toInt,rec1(3)) } ).toDF()

       
 

Things you need to know, when reading a Parquet & AVRO files. 

Once you get the DF out of the PARQUET or AVRO. You can access the values using the rec(INDEX)

Sample example:

       

val ordersDF = sqlContext.read.parquet("PATH")
ordersDF.map(rec => rec.getInt(0)+rec.getLong(1)+rec.getInt(2)+rec.getString(3)).take(12).foreach(println)

ordersDF.map(rec => rec(0)+"\t"+rec(1)+"\t"+rec(2)+"\t"+rec(3)).take(12).foreach(println)

import com.databricks.spark.avro._
val ordersDF = sqlContext.read.avro("CCA_Prep/avro/orders")
ordersDF.map(rec => rec(0)+"\t"+rec(1))

       
 


Things to remember when importing the data using the SQOOP.

1) Sqoop import MYSQL data to the TEXT file. 
       

sqoop import --connect "" --username "" --password --table order_items --as-textfile --target-dir --fields-terminated-by '' --lines-terminated-by ''



       
 

2) Sqoop import MYSQL data to the AVRO Data file.
       

sqoop import --connect "" --username "" --password "" --table order_items --as-avrodatafile --target-dir "" --fileds-terminated-by '' --lines-terminated-by ''


       
 

3) Sqoop import MYSQL data to the PARQUET Data file
       

sqoop import --connect "" --username "" --password "" --table order_items --as-parquetfile --target-dir "" --fields-terminated-by '' --lines-terminated-by ''

       
 




Wednesday, July 5, 2017

Java 8 Functional interfaces

Definition:

@FunctionalInterface annotation is useful for compilation time checking of your code.
You cannot have more than one method besides
static,
default and
abstract methods that override methods in Object in your @FunctionalInterface or any other interface used as a functional interface.

Insight 1:

Conceptually,
a functional interface has exactly one abstract method.
Since default methods have an implementation, they are not abstract.
Since default methods are not abstract you’re free to add default methods to your functional interface as many as you like.

Insight 2:

If an interface declares an abstract method overriding one of the public methods of java.lang.Object, that also does not count toward the interface’s abstract method count since any implementation of the interface will have an implementation from java.lang.Object or elsewhere.

Insight 3:

If an interface declares an abstract method overriding one of the public methods of java.lang.Object, that also does not count toward the interface’s abstract method count since any implementation of the interface will have an implementation from java.lang.Object or elsewhere.
e.g. Below is a valid functional interface even though it declared two abstract methods. Why? Because one of these abstract methods “equals()” which has signature equal to public method in Object class.
@FunctionalInterface
 interface MathOperation {
      int operation(int a, int b);
      @Override
      public String toString();  //Overridden from Object class
      @Override
      public boolean equals(Object obj); //Overridden from Object class
 }
While the intended use of Functional interfaces is for lambda expressions, method references and constructor references, they can still be used, like any interface, with anonymous classes, implemented by classes, or created by factory methods.