Running jmalloc on JavaFX Linux memory leak

Running a JavaFX application/Web Start application can causes a huge memory leak outside Java heap on Linux systems. My applications resident memory grow beyond 6 GB and continues, while at the same time JMC, JProfiler and other tools reports a heap size below 300Mb. The leak continues until the entire system becomes unresponsive.

Turning off hardware acceleration with -Dprism.order=sw is not an option available when using Java Web Start.

Demo application to reproduce the problem:

 (Code borrowed from: https://bugs.openjdk.java.net/browse/JDK-8161911)

import javafx.animation.Animation;

import javafx.animation.TranslateTransition;

import javafx.application.Application;

import javafx.scene.Node;

import javafx.scene.Scene;

import javafx.scene.control.Label;

import javafx.scene.layout.StackPane;

import javafx.stage.Stage;

import javafx.util.Duration;

public class Demo extends Application{

    @Override

    public void start(Stage primaryStage) throws Exception {

        Label l = new Label("A");

        StackPane sp = new StackPane(l);

        primaryStage.setScene(new Scene(sp, 200, 200));

        primaryStage.show();

        addTranslate(l);

    }

    private void addTranslate(Node target) {

        TranslateTransition tt = new TranslateTransition(Duration.millis(3000), target);

        tt.setByX(50);

        tt.setAutoReverse(true);

        tt.setCycleCount(Animation.INDEFINITE);

        tt.play();

    }

   

    public static void main(String[] args) {

        launch(args);

    }

}

Steps to run jemalloc/jeprof on demo application:

1. Save file to Demo.java
2. javac Demo.java
3. export LD_PRELOAD=/usr/lib64/libjemalloc.so.2
4. export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17
5. java Demo
6. jeprof --show_bytes --gif /usr/local/java/jdk1.8.0_144/bin/java jeprof*.heap >jeprof-demo.gif  

(* jemalloc steps borrowed from http://www.evanjones.ca/java-native-leak-bug.html)

(* On Suse: If you get an error indicating missing objdump:  zypper install binutils)

Result:

jmalloc result using Java 1.8.0_144

Real world application show a slightly different result (Now using Java 9+181):

This application uses a bit more time to fill up memory, but eventually makes the system
unresponsive.

System details:

Java version "1.8.0_144"
OS: openSUSE Leap 42.3

glxinfo |grep Open
Vendor: Intel Open Source Technology Center (0x8086)
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) Haswell Mobile
OpenGL core profile version string: 4.5 (Core Profile) Mesa 17.0.5
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.0 Mesa 17.0.5
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.1 Mesa 17.0.5
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10