Restrict CPU usage by java8 parallel stream/High CPU usage by Java 8 parallel stream

2017-06-17 18:42 RBanerjee imported from Stackoverflow

I was testing the performance of java 8 parallel stream api by creating a simple loop and adding elements in an array.

I am getting a huge performance boost over non parallel one.

But when I am checking my task manager I am seeing an uncontrolled CPU usage, it eats up all my CPU during that period.

here is my sample code,

    public static void withIfElse(boolean sorted) {

    int[] arr = new int[32768];
    Random rand = new Random();
    for(int i = 0;i < arr.length;i++)
        arr[i] = rand.nextInt(256);
    if(sorted)
        Arrays.parallelSort(arr);

    long nst = System.currentTimeMillis();
    long sum = 0;
    for(int j=0;j<100000;j++) {
        sum += Arrays.stream(arr).parallel().filter(m->m>=128).reduce((m,n)-> m+n).getAsInt();
    }

    System.out.printf("Time taken for boolean %b is %d ms. \n",sorted,(System.currentTimeMillis()-nst));

}

I am able to achieve almost 2x to 8x performance boost.

But ,

Cpu usage is uncontrolled. Is there any way to force java to only 2 cores ?

I have also tried setting

System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "1");

But still CPU usage is high.

Also is it recommended to use parallel stream?, because suppose in my application I have 10 user threads, or a web application where each request is a thread, in that case if I start a parallel stream on multiple threads which eventually eat all my cpu, there will be so many context switching.

2 answers

answered 2017-06-17 18:42 Joe C

Parallel streams run, by default, in the system-wide ForkJoinPool, which gives one thread for each core available on your machine.

If you want to change this behaviour, you can run your parallel stream in a ForkJoinPool of your choosing:

    ForkJoinPool fjp = new ForkJoinPool(2);
    ForkJoinTask<Long> sumFuture = fjp.submit(() -> {
        long sum = 0;
        for(int j=0;j<100000;j++) {
            sum += Arrays.stream(arr).parallel().filter(m->m>=128).reduce((m,n)-> m+n).getAsInt();
        }
        return sum;
    });

answered 2017-06-17 18:42 Eugene
My 0.02$: huge performance boost over non parallel one. huge is probably a bit too much here. And there might be good reasons for this - first, the way you test. It is so critical that there has been a question with over 500 up-votes especially for this: micro-benchmark. You might think you have good tests written, but jvm could have different plans for you.

For moderate data, indeed a sequential stream might be faster than a parallel one. It's a lot more work that has to be done by a parallel stream. But without actual measurements, it's close to impossible to say; but here are recommendations from the best in the field, like this.

When you say that the you want to limit java to some cores - the plain answer is that you can't; from java code that is. It has to be done from the OS, if it doable at all. What you have enable there via java.util.concurrent.ForkJoinPool.common.parallelism is limiting the threads, not the cores.

And a single thread can be split over multiple cpus - it depends on the Operating system itself.

`-509` vs `510`

I am seeing some kind of altered or erroneous data, with the use of JDBC. So I observe using H2 Database version 1.4.196 on Java 8 Update 151.

Here is a complete example.

Note how we retrieve the date value three times, first as a LocalDate object, secondly as text, and thirdly as an int year number extracted from a cast LocalDate object. In the text version we can see that the year is indeed negative. Mysteriously the LocalDate has a different year number and it is positive rather than negative. Seems like a bug.

private void doIt ( )
{
    System.out.println( "BASIL - Running doIt." );
    try
    {
        Class.forName( "org.h2.Driver" );
    } catch ( ClassNotFoundException e )
    {
        e.printStackTrace( );
    }

    try (
            Connection conn = DriverManager.getConnection( "jdbc:h2:mem:" ) ;  // Unnamed throw-away in-memory database.
    )
    {
        conn.setAutoCommit( true );
        String sqlCreate = "CREATE TABLE history  ( id IDENTITY , when  DATE ); ";
        String sqlInsert = "INSERT INTO history ( when ) VALUES ( ? ) ; ";
        String sqlQueryAll = "SELECT * FROM history  ; ";

        PreparedStatement psCreate = conn.prepareStatement( sqlCreate );

        psCreate.executeUpdate( );

        PreparedStatement psInsert = conn.prepareStatement( sqlInsert );

        psInsert.setObject( 1 , LocalDate.of( 2017 , 1 , 23 ) );
        psInsert.executeUpdate( );

        psInsert.setObject( 1 , LocalDate.of( -509 , 1 , 1 ) );
        psInsert.executeUpdate( );

        PreparedStatement psQueryAll = conn.prepareStatement( sqlQueryAll );
        ResultSet rs = psQueryAll.executeQuery( );
        while ( rs.next( ) )
        {
            long l = rs.getLong( 1 );  // Identity column.
            // Retrieve the same date value in three different ways.
            LocalDate ld = rs.getObject( 2 , LocalDate.class );  // Extract a `LocalDate`, and implicitly call its `toString` method that uses standard ISO 8601 formatting.
            String s = rs.getString( 2 );  // Extract the date value as text from the database using the database-engine’s own formatting.
            int y = ( ( LocalDate ) rs.getObject( 2 , LocalDate.class ) ).getYear( );  // Extract the year number as an integer from a `LocalDate` object.
            String output = "ROW: " + l+ " | " + ld + " | when as String: " + s+ " | year: " + y ;
            System.out.println( output );
        }

        conn.close( );
    } catch ( SQLException e )
    {
        e.printStackTrace( );
    }
}

When run.

ROW: 1 | 2017-01-23 | when as String: 2017-01-23 | year: 2017

ROW: 2 | 0510-01-01 | when as String: -509-01-01 | year: 510

So there seems to be something going on with JDBC being involved. Note how the year is presented as a positive 510 rather than a negative 509. I do not understand this behavior.

I can deduce that it is an issue within JDBC rather than within LocalDate. See this example code run live in IdeOne.com showing that a LocalDate object does indeed carry and report a negative year.

LocalDate ld = LocalDate.of( -509 , 1 , 1 ) ;
System.out.println( "ld.toString(): " + ld ) ;
System.out.println( "ld.getYear(): " + ld.getYear() ) ;

Notice how we do not get converted from -509 to 510 when dealing with a LocalDate only, with no JDBC.

ld: -0509-01-01

ld.getYear(): -509

How to calculate key through the message and the generated Mac

I'd like to know if there is a way to get a key through the message and the MAC generated by applying an algorithm for decryption (which is SHA1) to the message along with the key, but the key does not know it, how could I do? I have an example that computes the MAC randomly, which is as follows:

public class MACApp {
    public static void main(String[] args) {
        //if(args.length < 1) System.out.println("Úsese MACApp \"hola\"");
        //else performMACTest(args[0]);
        performMACTest("Hola");
    }

    public static void performMACTest(String s) {
        try {
            // Elegimos el algoritmo de hashing

            String alg = "HmacMD5";
            // Creamos un objeto Mac que implementa el algoritmo especificado
            Mac mac = Mac.getInstance(alg);
            // Construimos un genrador de claves privadas
            KeyGenerator generador = KeyGenerator.getInstance(alg);
            // Creamos una clave privada con el generador de claves

            SecretKey clavePrivada = generador.generateKey();
            mac.init(clavePrivada);
            // Pasamos la cadena de caracteres a un array de bytes y lo procesamos
            // con el algoritmo del Mac, es decir, procesamos (de manera indirecta) el
            // string que hemos pasado como parámetro
            mac.update(s.getBytes());
            // Indicamos que ya hemos acabado y recogemos el resultado
            byte[] b = mac.doFinal();
            System.out.println("\nMensaje original: " + s);
            System.out.println("\nMAC    :" + Conversor.aCadenaDeHexadecimales(b));
        } catch (Exception e) {
            System.out.println(e.getMessage());
        }
    }
}

class Conversor {
    public static String[] digitos =
            {"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "A", "B", "C", "D", "E", "F"};

    public static String aCadenaDeHexadecimales(byte[] array) {
        String cadena = "";
        for (int i = 0; i < array.length; i++)
            cadena = cadena + convertirUno(array[i]);
        return cadena;
    }

    private static String convertirUno(byte valor) {
        int n = valor;
        if (n < 0) n = 256 + n;
        int d1 = n / 16;
        int d2 = n % 16;
        return digitos[d1] + digitos[d2];
    }
}

If someone I could help find the key. I put an example:

Message: 231456789 487654 500

MAC: 5807a9647ed99e98b8d197e608c1de8bcffa571f

The type of Hash to encrypt/decrypt is SHA1 and also the size of the key is 24 bits, i.e. must have 3 characters.

Java8: Is it a good practice to always create methods instead of using lambda?

As saw here Java8: HashMap<X, Y> to HashMap<X, Z> using Stream / Map-Reduce / Collector, it seems to be more performant to use method rather than lambda style.

By example :

Map<String, String> x;
Map<String, Integer> y =
    x.entrySet().stream()
        .collect(Collectors.toMap(
            Map.Entry::getKey,
            e -> Integer.parseInt(e.getValue())
        ));

should be more performant than

Map<String, String> x;
Map<String, Integer> y =
    x.entrySet().stream()
        .collect(Collectors.toMap(
            e -> e.getKey(),
            e -> Integer.parseInt(e.getValue())
        ));

So my question is : Is it a good practice to always create methods instead of using lambda ?

With the same example :

private Integer transformValue(Entry<String, String> e) {
    return Integer.parseInt(e.getValue())
}


Map<String, String> x;
Map<String, Integer> y =
    x.entrySet().stream()
        .collect(Collectors.toMap(
            Map.Entry::getKey,
            MyClass::transformValue
        ));

Collect List into a Map<Object1 , Map<Object2, Object3>> in Java 8

I want to iterate a list of objects. Each of object contains some variables of objects. More specifically:

public class KCenterDocumentMapping {
    private DigitalFileCategory digitalFileCategory;
    private DocumentType documentType;
    private KCenterCategory kCenterCategory;
    private KCenterFolder kCenterFolder;
    private KCenterDocumentType kCenterDocumentType;

    //setters & getters
}

My list is:

List<KCenterDocumentMapping> allMappings = db4.retrieveMapping();

The desired Map Structure is:

Map<DigitalFileCategory, Map<DocumentType, KCenterDocumentMapping>> kCenterDocumentMappingMap = new HashMap<>();

In my current implementation I received "non-static method cannot be referenced from a static context".

Map<DigitalFileCategory, Map<DocumentType,KCenterDocumentMapping>> grouping = kCenterDocumentMappingList.stream()
                .collect(Collectors.groupingBy(KCenterDocumentMapping::getDigitalFileCategory,
                        Collectors.mapping(KCenterDocumentMapping::getDocumentType, this)));

Any ideas?

How to get Kth largest element of a DoubleStream in java

What's the best way of getting the Kth largest element in a DoubleStream in java?

I know we can do .max().getDouble() to get the largest element.
Functionally parsing/assembling a file
I have a sql file that I am trying to parse. I can easily load the whole file, string.split(";") and properly delimit the file. However, from a challenge and learning standpoint I would like to apply some functional foo and some streams to handle a file of arbitrary size. But that, gets a lot trickier.

Essentially I need to:
1. Load an arbitrary buffer of data (let's say, a line since I can use Java's BufferedReader to stream those
2. Continue to concatentate those line until there is a delimiter ;
3. Emit the concatenated result
4. Start a new sql command after the delimiter
5. Final result: List<String> representing all sql commands to execute
Here is a sample:
```
CREATE SCHEMA HelloWorld;

USE HelloWorld;

CREATE TABLE HelloWorldTest (
  id int PRIMARY KEY NOT NULL AUTO_INCREMENT,
  message VARCHAR(32)
);

INSERT INTO HelloWorldTest (message) VALUES ('Hello world!');
INSERT INTO HelloWorldTest (message) VALUES ('this is a test!');
```
Inefficient Java to do the job:
```
 String fileData = IOUtils.toString(sqlFile);
 List<String> sqlStats = stream(fileData.split(";")).filter(s -> s.length() > 0)
        .collect(toList());
```
How can I do this with stream and functional programming to handle any size file? Non-java answers are welcome as Java may not be able to do the job with the native libraries

Related questions

Restrict CPU usage by java8 parallel stream/High CPU usage by Java 8 parallel stream

2 answers

See also questions close to this topic

-509 vs 510

`-509` vs `510`