hi, it's not easy to tell where the problem is coming from. you should get some decent results leaving all those constants at their default values.
Just to doublecheck.. how are you inputting the raw sensor values to the algorithm? I don't know if you noticed that the LSM9DS1 measures the values with three different reference systems, one for each sensor,
from the datasheet, page 10
The kalman filter of course expects all the values to be expressed in the same reference system.
What we do is convert all of them to a right handed, Zup up like the one Blender uses. In order to do that we map the raw values like this:
kalman.set_gyro(imu->gy, imu->gx, imu->gz);
kalman.set_acel(-imu->ay, -imu->ax, -imu->az);
kalman.set_mag(imu->my, -imu->mx, imu->mz);
Note: the accel reference system might seem equal to the gyro one, but you are getting the inertial or Fictitious force which points in the opposite direction from the actual acceleration vector. More info in this article