Related questions

Restrict CPU usage by java8 parallel stream/High CPU usage by Java 8 parallel stream

2017-06-17 18:42 RBanerjee imported from Stackoverflow

I was testing the performance of java 8 parallel stream api by creating a simple loop and adding elements in an array.

I am getting a huge performance boost over non parallel one.

But when I am checking my task manager I am seeing an uncontrolled CPU usage, it eats up all my CPU during that period. Check the CPU usage

here is my sample code,

    public static void withIfElse(boolean sorted) {

    int[] arr = new int[32768];
    Random rand = new Random();
    for(int i = 0;i < arr.length;i++)
        arr[i] = rand.nextInt(256);
    if(sorted)
        Arrays.parallelSort(arr);

    long nst = System.currentTimeMillis();
    long sum = 0;
    for(int j=0;j<100000;j++) {
        sum += Arrays.stream(arr).parallel().filter(m->m>=128).reduce((m,n)-> m+n).getAsInt();
    }

    System.out.printf("Time taken for boolean %b is %d ms. \n",sorted,(System.currentTimeMillis()-nst));

}

I am able to achieve almost 2x to 8x performance boost.

But ,

Cpu usage is uncontrolled. Is there any way to force java to only 2 cores ?

I have also tried setting

System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "1");

But still CPU usage is high.

Also is it recommended to use parallel stream?, because suppose in my application I have 10 user threads, or a web application where each request is a thread, in that case if I start a parallel stream on multiple threads which eventually eat all my cpu, there will be so many context switching.

2 answers

  • answered 2017-06-17 18:42 Joe C

    Parallel streams run, by default, in the system-wide ForkJoinPool, which gives one thread for each core available on your machine.

    If you want to change this behaviour, you can run your parallel stream in a ForkJoinPool of your choosing:

        ForkJoinPool fjp = new ForkJoinPool(2);
    ForkJoinTask<Long> sumFuture = fjp.submit(() -> {
        long sum = 0;
        for(int j=0;j<100000;j++) {
            sum += Arrays.stream(arr).parallel().filter(m->m>=128).reduce((m,n)-> m+n).getAsInt();
        }
        return sum;
    });

  • answered 2017-06-17 18:42 Eugene

    My 0.02$: huge performance boost over non parallel one. huge is probably a bit too much here. And there might be good reasons for this - first, the way you test. It is so critical that there has been a question with over 500 up-votes especially for this: micro-benchmark. You might think you have good tests written, but jvm could have different plans for you.

    For moderate data, indeed a sequential stream might be faster than a parallel one. It's a lot more work that has to be done by a parallel stream. But without actual measurements, it's close to impossible to say; but here are recommendations from the best in the field, like this.

    When you say that the you want to limit java to some cores - the plain answer is that you can't; from java code that is. It has to be done from the OS, if it doable at all. What you have enable there via java.util.concurrent.ForkJoinPool.common.parallelism is limiting the threads, not the cores.

    And a single thread can be split over multiple cpus - it depends on the Operating system itself.

See also questions close to this topic

  • How can I automate a project developed in Swift?

    How can I automate project developed in Swift?

  • Vaadin Window resize and component inside auto alignment

    I have a class that extends Window as follows. I am only giving Window related code. Window Component code is removed to keep the code and question clear. I got a window looks like in the picture. When I resize the window, I was expecting that the components inside the window will also be resized. I guess there is some API call I am missing to maintain the component ration increased together with the window. Or do I need to do it by myself programmatically? Please let me know how it can be done? 

     public class VisibleColumnPanel extends Window implements Window.ResizeListener{

  public VisibleColumnPanel(){

    center();
    setResizable(true);
    setModal(true);
    setImmediate(true);
    //setSizeFull();
    addListener(this);
    this.setWidth(720, Unit.PIXELS);
    this.setHeight(600, Unit.PIXELS);
 }

@Override
public void windowResized(ResizeEvent resizeEvent) {

    System.out.println("hit here now");

}


}

    Normal window without maximizing it:- enter image description here

    After maximizing the window:- enter image description here

    So is there any way that the component inside the window all also expend or maintain their ration together with the window?

    I could programmatically do it if I could handle maximize button click event. I implemented implements Window.ResizeListener. windowResized event only fired when I try to resize the window by dragging the mouse. Is there any way to get maximize/restore button click event?

  • Hash a whole object without converting to byte[] first

    I want to get sha2 hash of particular java object. I don't want it to be int, I want byte[] or at least String. I've got the following code to create sha2:

    static byte[] sha2(byte[] message) {
    if (message == null || message.length == 0) {
        throw new IllegalArgumentException("message is null");
    }
    try {
        MessageDigest sha256 = MessageDigest.getInstance(SHA_256);
        return sha256.digest(message);
    } catch (NoSuchAlgorithmException e) {
        throw new IllegalArgumentException(e);
    }
}

    I can just convert my object to byte[], but I don't think it's a good idea to store big array in memory just to create 32 byte array. So how can I compute sha2(or maybe another crypto hash function) of object?

  • Java 6 to Java 8 Migration Issue

    I have tried to update legacy code from java 6 to java 8 by updating pom (maven-compile-plugin source and target and javax javaee-web-api.

    When I change the server java to jdk 8 and try to run it I get the following error:

    Exception while invoking class org.glassfish.ejb.startup.EjbDeployer load method java.lang.RuntimeException: EJB Container initialization error at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:234)

    ...

    Caused by: java.lang.IllegalAccessException: Class com.sun.ejb.containers.BaseContainer can not access a member of class my.package.$Proxy280 with modifiers "public"> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)

    Is there any code which I need to update to fix the above error?

  • Ant javac source/target options not working

    I recently upgraded the java version for my application. When I ran my ant script which had javac set as:

    <javac destdir="${build.dir}" target="${javac.target}" source="${javac.source}" debug="true" nowarn="${javac.nowarn}" deprecation="${javac.deprecation}" optimize="${javac.optimize}" encoding="${javac.encoding}" failonerror="true">
     <src path="${build.dir}" />
     <classpath refid="master-classpath" />
</javac>

    When I run the the script with this, the project does not build properly.

    So I changed it to:

    <javac destdir="${build.dir}" target="1.8" source="1.8" debug="true" nowarn="${javac.nowarn}" deprecation="${javac.deprecation}" optimize="${javac.optimize}" encoding="${javac.encoding}" failonerror="true" sourcepath="">
    <src path="${java.src.dir}" />
    <classpath refid="master-classpath" />
</javac>

    And ran the script, I get the error:

     [javac] javac: invalid target release: 1.8
 [javac] Usage: javac <options> <source files>
 [javac] use -help for a list of possible options
  • map row in Dataset<Row> to object class Spark Java

    Working with HBase and Parquet, I have written code to get values from HBase and map values to Object class but I am having trouble replicating this with Parquet using Dataset.

    HBase:

    JavaPairRDD<ImmutableBytesWritable, Result> data = sc.newAPIHadoopRDD(getHbaseConf(),
            TableInputFormat.class, ImmutableBytesWritable.class, Result.class);

JavaRDD<List<Tuple3<Long, Integer, Double>>> tempData = data
                .values()
                //Uses HBaseResultToSimple... class to parse the data.
                .map(value -> {
                    SimpleObject object = oParser.call(value);
                    // Get the sample property, remove leading and ending spaces and split it by comma
                    // to get each sample individually
                    List<Tuple2<String, Integer>> samples = zipWithIndex((object.getSamples().trim().split(",")));

                    // Gets the unique identifier for that sp.
                    Long sp = object.getPos();

                    // Calculates the hamming distance for this sp for each sample.
                    // i.e. 0|0 => 0, 0|1 => 1, 1|0 => 1, 1|1 => 2
                    return samples.stream().map(t -> {
                        String alleles = t._1();
                        Integer patient = t._2();

                        List<String> values = Arrays.asList(alleles.split("\\|"));

                        Double firstA = Double.parseDouble(values.get(0));
                        Double second = Double.parseDouble(values.get(1));

                        // Returns the initial sp id, p id and the distance in form of Tuple.
                        return new Tuple3<>(snp, patient, firstAllele + secondAllele);
                    }).collect(Collectors.toList());
                });

    I read data from Parquet into Dataset but simple can't replicate above approach. 

    Dataset<Row> url = session.read().parquet(fileName);

    I just need to know how to map the rows in Dataset<Row> to object class as I do with .map(value -> {...in the above approach.

    Any help would be appreciated.

  • Did Eclipse Collections get deprecated by Java 8?

    I recently stumbled across Eclipse Collections, and they look awesome - pretty much about as awesome as Java 8 streams looked. I read through a few introductions, presentations and tutorials, and it seems like pretty much everything EC added, you can do with streams nowadays.

    Without in any way meaning to put down EC, is there any value the library still adds now that we have streams, that maybe got glossed over in what I read ? Or did it essentially go the way of Joda time; being so good that it got adopted into Java pretty much verbatim negating the need for the library ?

  • Java Stream API map argument

    I have some misunderstanding about Java 8 static method references.

    This is a correct statement:

    Stream.of("aaa", "bbb", "cccc").map(String::length);

    AFAIK map needs a Function<T, R> interface as an argument with method similar to:

    R apply(T t);

    But method length() of String class doesn't accept any arguments:

    public int length() {
    return value.length;
}

    1) How does it correlate with apply method which needs an argument T t?

    2) If I write String::someMethod doesn't it mean that someMethod should be static? Because I call it by class name, not object.

    Thank you!

  • Summing up BigDecimal in a map of list of objects in Java

    I'm stuck with some elegant ways to get a summation of BigDecimals in a map. I know how to calculate the sum in a map of BigDecimal but not a List of object with a BigDecimal.

    The structure of my objects are as below:

    Class Obj {
  private BigDecimal b;
  // Getter for b, say getB()
}

Map<String, List<Obj>> myMap;

    I need to get a sum of all bs in myMap. Looking for some elegant ways to do this in Java, may be using streams?