Test #1

AxonSpark

So I went and got some new 6p6c RJ12 headers and used the wiring from an Ethernet cable to make the new cables. The process went smoothly, and I was able to power up the whole suit and initialize the default biped configuration.

However, things went less smoothly inside Blender 2.79. Since I was using myself as the test subject, I stepped away from my laptop, faced south, used my wireless mouse to hit the 'Receive' button, and then went into a T-Pose which I maintained for several seconds to let the sensors settle. In order to get out of the calibration and not break the T-Pose, I used the voice dictation feature in OSX to hit the Return key when I issued a verbal command. Here's a video of the results:

Keep in mind that I was using Blender's internal screencast feature (need to find a better solution), and then slowed down the image sequence to 20% speed in Premiere, so the T-Pose is being held longer than it might seem in the video.

I'm not sure what the problem is, but the joints on the rig are not bending the same way I am (for instance lifting a leg and bending my knee forward results in completely different result in Blender, usually bending to the side, or sometimes backward), and even when the rig goes in roughly the right direction the bend doesn't correspond to the degree that I'm bending (eg., I'm being my arms a lot more than the rig is).

Any ideas on what might be going wrong? I know I last individually calibrated the K-Ceptors a while ago, but I would think that the calibration data would still be valid.

daylanKifky

AxonSpark

Hey
A shout-out to you for crimping the new RJ12 headers and for doing the one-man-band capture!! the voice command technique in quite ingenious!

The behaviour you are getting on the model looks a lot like the performer (in this case yourself, but let's just call it that way) was facing in the wrong direction. But as you mentioned, you were facing south... 🤔

I would take you sensors for correctly calibrated, even if you did it several months ago. You usually can tell when you are wearing a non calibrated sensor because it drifts even if you are completely steady. And it doesn't seems the case.

It's hard to tell what's going on from the video, but there's something that i'm having trouble understanding. During pose-calibration, the sensors on the back of the body normally(*) have their Z-axis pointing to the +X of the scene. And those on the legs should normally point to the -X on the scene. But during your calibration some of them were pointing in direction of the Y axis.

Do you have any idea of why is this happening?

(*) This doesn't have to necessarilly be like this. It depends on how the K-Ceptors are positioned on the body. This applies if the KCs on the back are resting on the flat over the spine area, and the ones on the legs are resting flat on the front of your legs.

daylanKifky

Hi!
I came out with a workflow to debug what's happening.

You can put together a simplified rigid "performer" to better see the orientation of the KCs in blender. Just use elastic bands or tape to fix three KCs to a rigid support (like a long ruler). You should also fix the SBC, Hub and battery.
All three KCs should be oriented roughly in the same direction

Then take a look at this video:

Files for this test:

Let me know how it goes.

PS: in the video we ask you to point the KCeptors north. This is because they are supposed to be on the back of our dummy performer, so it is pointing south

AxonSpark

Hi @daylanKifky,

Here are my test results:

So, I'm definitely having drifting/orientation issues (or certain axes being linked to incorrect corresponding axes). I'm guessing I'll have to re-calibrate, correct? If I do, I think I may need to update my software as well, since I'm currently running Notochord v0.1.1b.

I'm pretty sure I set up the spine rig correctly, but just to make sure, here's a picture of my set up:

As far as your question about K-Ceptors pointing along the wrong axes goes, I'm not sure why this is happening. Generally, I orient the K-Ceptors along the input/output lines so the ports from one K-Ceptor are pointed toward the next. Do they need to be oriented differently? As for the back, the bottom of the sensors faces towards the back, and the top of the sensors faces away from the back. I drew some info on your DefBypedConf-IDModules to try and illustrate some of this:

Let me know how I should proceed. Thanks!

Note for anyone else doing this test: your K-Ceptors need to be plugged into CH 1 on the Hub (unless you change the XML file).

daylanKifky

Yep, it looks like you have a sensor calibration problem there. I remember you were having good results on the last session, some months ago. That calibration should stay on the sensors, so I have no idea what's the cause of the current problem.

I think trying a re-calibration now is a good idea. I suggest you to take a set o three KCs, calibrate each one, and then use them on the spine test.

(There wasn't many visible changes since v0.1.1b, we are working on modifying some internal interfaces, but basically the functionality of the program is the same)

AxonSpark I'm pretty sure I set up the spine rig correctly

You did

AxonSpark I orient the K-Ceptors along the input/output lines so the ports from one K-Ceptor are pointed toward the next. Do they need to be oriented differently? As for the back, the bottom of the sensors faces towards the back, and the top of the sensors faces away from the back.

In technical terms, you can orient the KCs as you want, as far as they stay well-fixed to to body part they have to capture. In this case I was asking it in order to confirm if the differences in orientation I noticed in your first video were due to a sensor malfunction (perhaps you were just positioning them that way)

The arrangement you are describing is the standard one.

AxonSpark

Hey daylanKifky,

Just dropping a note to say that more test results are forthcoming. I did the K-Ceptor calibrations today, along with test cube and spine tests and things seem to be working. I also decided to do another full test, and things are generally bending in the right direction. However, the movements of the rig still don't fully correspond to what I'm doing and seem pretty stiff, and it seems to take a while for parts of the rig to get into a pose when I make one.

I have the graph images generated by each calibrated K-Ceptor, pictures taken of how I have the sensors arranged on the suit, and I did a simultaneous screen cap/video of the full biped test. All I have to do now is redo the spinal tests so I can record them. It might take me a day or two due to work stuff, but it's coming. :-)

In the meantime I made a few suggested changes to the Wiki.

daylanKifky

AxonSpark

Many thanks for taking your feedback so seriously! I approved your edits on the wiki. It was a good idea to let me know about your edits here: we receive an annoying amount of spam though wiki articles editing. So it hard to tell from legitimate edits at first sight.

AxonSpark it seems to take a while for parts of the rig to get into a pose when I make one.

Do you mean that you are getting a noticeable lag when using the full body, that is not present on smaller tests?

AxonSpark

daylanKifky Yes, there is a significant amount of lag in the full body that is not present in the smaller tests.

AxonSpark

Ok, here we go. I have a lot of material, so I'm going to parse it out into 3 posts. Here are the results from the latest round of K-Ceptor calibrations.

AxonSpark

After I calibrated the K-Ceptors, I tested each one with the test cube Blender file. All 15 seemed to be working properly. I also performed spine tests after the cube tests, and they too seemed to be giving me good results.

However, when I did the spine tests again today in order to record them, one of the K-Ceptors didn't seem to be working right. For some reason, K-Ceptor 1 in Group 4 had a constant drift that didn't occur on the previous test. Do you know what might have caused this? Since I re-calibrated and tested all the K-Ceptors 3 days ago, they should all be working correctly. I can't imagine what would have caused this one to malfunction (note: I did the test with Group 4 a couple times to confirm the result, and in both instance K-Ceptor 1 was drifting).

Here are the full results of the spinal tests (apologies for the framing of the video in the corner):

AxonSpark

Here is my second full biped test:

I did the test a couple times before filming/capturing and got the same results you see here. With my Attempt #1, the joints didn't bend in the right direction, but when I reloaded the Blend file and tried again, the joints would bend correctly.

The other big issue is some sort of lag. The full rig is nowhere near as responsive as the test cube or spinal test. When I move, the corresponding parts of the rig sometimes sort of go in the right direction, but the movements are much more subtle than the ones I'm making. It also seems to take the rig a long time to match my pose. For instance when I lifted my arms over my head, the arms on the rig gradually drifted up to match, and the rig arms weren't very responsive when I put my arms back down. It seems like rig is reacting in time with my movements, but the data is being applied in an odd, slow way.

A few things to note:

These issues were happening before doing the screen capture, so I don't think it was an issue of system resources.
At this point K-Ceptor 1 in Group 4 was working fine.
Despite some strap issues with the right thigh and left arm, the sensors were all in place when I started.

Also, to give you a better view of how I setup my suit, I had a friend take some pictures:

I think I'm ultimately going to have to get something better than this jogging harness. Do you happen to know what the testers in India have been using?

Lastly, I've noticed that the bone that corresponds with the base doesn't move (most noticeable when I was doing squats and push-ups) and everything sort of rotates around it. In the future will it be able to move around in 3D space?

daylanKifky

Hi!
thanks as always for your accurate report. It might take me a couple of days to process the information.

AxonSpark Do you happen to know what the testers in India have been using?

We will ask. Looks like a pretty neat solution

AxonSpark Lastly, I've noticed that the bone that corresponds with the base doesn't move (most noticeable when I was doing squats and push-ups) and everything sort of rotates around it. In the future will it be able to move around in 3D space?

Yes it will, in fact you can already export that information when re-transmiting your capture live by selecting the option Send root bone height. We will make it available on the Blender viewport as soon as possible.

AxonSpark I logged into the Wiki to add a section on connecting to your WiFi router for testers that might be new to the Raspberry Pi, but it says that I was blocked indefinitely in June. Not sure why.

Sorry about that, we receive spam accounts everyday and just block them all. It is hard to tell them from real ones, since it's an automated process. It should be working now

AxonSpark

P.S. I logged into the Wiki to add a section on connecting to your WiFi router for testers that might be new to the Raspberry Pi, but it says that I was blocked indefinitely in June. Not sure why.

daylanKifky

Hi @AxonSpark
Sorry for the delay on my response. The behavior you are getting is inconsistent, and contradictory (i guess you already figured out that part 😅), so I needed some time to carefully go though it.
I'm still not able to give you a definitive response, but ..

Let's review what we can assume at this point.

The K-Ceptors seems to be working much better on your spine test 1.2, the behaviour you are getting once you enter capture mode is coherent.

The behavior on the second attempt of your full body test seems to be correct, in the sense that the joints are bending on the right direction. But the lag is huge, so we can't be 100% sure of this statement.

The origin of the lag is a complete mystery. You were using the exact same hardware, software, and network to make the spinal tests (I assume) and they presented no lag. Some probable causes i can think of are:

Network congestion: The amount of data coming from the Chordata system is not much, but under certain conditions the packets might accumulate causing significant delays, specially under wifi. This is very unlikely to happen, specially if you are using a decent access point. But just to be sure: Could you test it with a LAN cable instead of the WIFI? When doing this be sure to turn off the WIFI connection on your raspberry or router (not only your pc), otherwise you might end up transmitting over wifi even if the LAN cable is connected.
Pose-processing delays: This is a (again) a very unlikely scenario. Your computer is probably able to handle the processing of the capture, the fact that the spinal test runs smoothly confirms that. Both files are equivalent in terms of bone and poly count. But perhaps there's something on the configuration of the default_biped file that's causing the lag. An example would be debug mode turned on. On that mode Blender prints every received message to the console, which is a blocking operation. Try turning everything but the strict necessary off under Record, Transmission and Advanced panel.

The still open questions are:

Why the default_biped test worked only after reloading the file? Have you noticed something similar before? This looks like a bug on the add-on. It would be great if we can reproduce the exact situation were it happens. What did you did on Blender before you made that attempt?
Why the K-Ceptors lost their calibration? I assume they were stored properly. Perhaps the temperature on your flat is now drastically different from the time you first calibrated them?
What's happening with K-Ceptor 1 in Group 4? That intermittent behavior you are describing seems to indicate some kind of bad contact on a component on the KCeptor. The U1 and U2 shouldn't be involved (otherwise you would get a communication error) but perhaps there's a badly soldered pad on some of the capacitors around U1.
Another possible bad soldering might be some pad on U3 (the memory). If you ever get this behavior again try running notochord with a higher verbosity level, in order for it to print warnings (such as a memory not found)

PS: we still had no answer from the betatesters in india. @robin is that you on the photo @AxonSpark is talking about?

AxonSpark

Hey @daylanKifky,

I completely understand the delay. Between Maker Faires, the upcoming Blender conference, and developing new K-Ceptors, this is probably a busy month for the Chordata team. To answer your quesitons:

daylanKifky You were using the exact same hardware, software, and network to make the spinal tests (I assume) and they presented no lag

Yes, everything was the same.

Network Congestion

I also think this is unlikely, as in the past 4 people have shared the connection on my Asus Dark Knight router without problems. And when I run the tests I’m alone in my apartment, so I’m the only one using it. I have some freelance editing work this week, but I’ll get the required cables to test the default biped configuration over Ethernet as soon as I’m able.

Pose-processing delays

I’m not using debug mode. Although, here’s a screenshot of the panels as they appear after I load the default_biped blend file (I haven’t touched these when doing testing. Just the Connect and Receive buttons):

Also, I’m currently using 2.79b if that helps.

daylanKifky Why the default_biped test worked only after reloading the file? Have you noticed something similar before? This looks like a bug on the add-on. It would be great if we can reproduce the exact situation were it happens. What did you did on Blender before you made that attempt?

I have not noticed anything similar before, and I didn’t do anything special in Blender. My order of operations is this:

Turn on the power and give my Raspberry Pi a moment to boot up
Connect to the Pi via the MacOS terminal using SSH
Run Notochord with the default_biped.xml file
Open up default biped blend file and connect to the Pi
Do my wireless mouse/voice command combo to start the T-Pose calibration and capture
The joints don’t bend correctly, so I reload the default_biped blender file again, and for some reason it works

daylanKifky Why the K-Ceptors lost their calibration? I assume they were stored properly. Perhaps the temperature on your flat is now drastically different from the time you first calibrated them?

What is considered proper storage? Since the suit has a lot of parts, I store everything in a small canvas duffel bag that sits on my plastic lighting case on my equipment shelf which is made of metal. Also, the temperature in my apartment hasn’t drastically changed. Athough, I should ask: what sort of temperature fluctuation would cause the calibration to fail?

daylanKifky What's happening with K-Ceptor 1 in Group 4? That intermittent behavior you are describing seems to indicate some kind of bad contact on a component on the KCeptor. The U1 and U2 shouldn't be involved (otherwise you would get a communication error) but perhaps there's a badly soldered pad on some of the capacitors around U1.
Another possible bad soldering might be some pad on U3 (the memory). If you ever get this behavior again try running notochord with a higher verbosity level, in order for it to print warnings (such as a memory not found)

I looked at the connections under a magnifying glass, and everything seems to be soldered well. Here is a screen shot of me testing this K-Ceptor in verbose mode:

Hope some of this helps.

daylanKifky

AxonSpark

yeah you are right, this is a really busy month. We are also preparing the production for the second beta testing program.

Regarding your situation: As I supposed you do have a good router, so the network congestion hypothesis is even more unlikely. Still trying the Ethernet cable will help discard the WIFI factor.

It seems like your configuration is clean.

Is the procedure you are describing exactly the same each time? Do you have to necessarily repeat this step each time?

AxonSpark The joints don’t bend correctly, so I reload the default_biped blender file again, and for some reason it works

If that's the case then something really strange is happening. But at least we have identified a procedure to reproduce it.

From the dump you are sending it looks like the calibration stored on the EEPROM was correctly read. (you can tell by the word EEPROM next to each calibration vector/matrix. I guess it was working fine that time.

I guess at this point the only thing we can try is to expand the spinal test, to see how a slightly bigger set of sensors behave. For example you can build an expanded rigid performer with which you can test the functionality of the joints more comfortably. Something like this:

What do you think? You will need a different .blend and XML for that, I'll send them as soon as possible

AxonSpark

@daylanKifky Sounds good to me.

To answer your question about the procedure, yes it is the same each time, and thus far I have always needed to reload the default_biped blend file to get the joints to bend correctly.

Also, I recently scrapped an old DIY tripod dolly, so I have plenty of wood on hand to make the expanded rigid performer.

In thinking about the environment where I'm doing this testing, I'm wondering if my refrigerator might be causing interference. I recently came across a YouTube video critiquing the Perception Neuron (which I believe uses similar sensors), and this was one of the things that was mentioned. My apartment is a little odd in that since my kitchen is a weird triangle shape, the fridge is in the dining room/office area where I typically work/do the tests. However, when I perform the spine tests I'm sitting about 5-6 feet from the fridge and there are no problems. I'm about 1-2 feet closer to it when I'm doing the full biped tests. Might this also be affecting results? If so, I can try doing the test in another room.

Also, would my wireless mouse be causing problems? After issuing my voice command to press the 'return' key to start receiving data, I have to take a moment to lean over and move the mouse cursor off the buttons so I can get into pose mode with the next voice command, but I always give the sensors plenty of time to settle back into the T-Pose. Would this be causing an issue?

daylanKifky

AxonSpark To answer your question about the procedure, yes it is the same each time, and thus far I have always needed to reload the default_biped blend file to get the joints to bend correctly.

OK
But, this issue is not present with on the spinal test, right??

AxonSpark In thinking about the environment where I'm doing this testing [...]

Electric motors like the one on fridge do generate an electromagnetic field. But I'm not sure it's enough to create such a massive interference on the whole capture.
The best to tell is be to make a test. You can put the complete hierarchy of sensor to work without even wearing it. Just connect them all as if they were on your body, but just place them over a table, or over your dummy performer. Place them on different parts of your apartment and try to find if there's some particularly problematic spot.
You can leave all the KCs connected, and try the default_biped and the spinal test on each spot to see if you have different results.

AxonSpark Also, would my wireless mouse be causing problems?

The movement you do with your arm might cause some calibration inconsistency on that particular arm. But we are facing a much bigger problem right now. And as you correctly guessed, giving it some time to settle should reduce the inconsistency.

Files for extended dummy performer

only spine and legs blend file
onlu_spine_and_legs.xml

The connections should be
Spine:
CH_1 ---> KC#0 ---> KC#1 ----> KC#2
Left_leg:
CH_4 ---> KC#0 ---> KC#1
Right_leg
CH_6 ---> KC#0 ---> KC#1

Yet another test

Another (unlikely) possible problem that might be occurring is the Raspberry getting overload. The notochord on normal execution with a default_biped should use no more than 15% of one of the raspberry's cores. You can test this with a program called htop

First install it on your raspberry with

sudo apt install htop

While you are performing the tests open a second terminal window, access the RPi though SSH and open htop to monitor the processor and memory usage of all processes (notochord will probably be on top, if not you can search for it with F3)

AxonSpark

daylanKifky OK
But, this issue is not present with on the spinal test, right??

Correct, this issue is not present when I conduct the spinal tests.

AxonSpark

Hey @daylanKifky,

I did more tests today, and will be posting videos of them in the next day or two, but basically this is what happened:

1 - Re-calibrated K-Ceptor 1-4

2 - The spine & leg tests went mostly well. Rotations/movements were mostly correct, though they seemed off in some instances (I think. It's hard to tell with the wireframe) and there seemed to be no lag.

3A - I did some full biped tests to see if either the Pi or the router was experiencing some sort of heavy load. I was experiencing the same lag problems as before, and there wasn't much CPU usage from the Norochord program on the Pi and the transmitted data didn't seem to be taking up much bandwidth on my router.

3B - I re-did the test in another room (which was closer to the router) and got the same lagging results as before.

3C - This time around the joints were not bending in the right direction, and reloading the full biped blend file had no effect, nor did resetting the Pi and loading the blend file again. Each time there were erroneous rotation directions (most notably the knees bending backward).

Notes:

In both the spine/leg and full biped tests, Notochord's CPU usage on the Pi tended to be between roughly 5% and 7.5%.

I was not wearing my belt, nor did I have my keys or phone in my pocket during the full biped tests in order to rule them out as sources of interference.

daylanKifky

AxonSpark 2 - The spine & leg tests went mostly well. Rotations/movements were mostly correct, though they seemed off in some instances (I think. It's hard to tell with the wireframe) and there seemed to be no lag.

I also noticed this on your first spinal test video, but thought it was just my perception of it. I looked like when you tilted your rigid performer on a given direction, the digital model rotated on a slightly different heading angle.

Perhaps using the upper-right viewport on the spinal .blend to find the south will solve this:
Before doing the proper pose calibration do a fake one, and just use it to find the direction in which the Z axis of the spinal sensors align with the X axis of the scene, like it's shown on the minute 1:02 of our video. Mark that direction on the room and use it to do all pose calibrations.

AxonSpark 3A, 3B, 3C

everything seems to indicate that there's some sort of problem with the default biped blend file, which is awkward because the spinal test file is just a modification of this one, and the lag wasn't present on the first video you posted on this thread.
In order to confirm it we should expand the spinal test blend file to use all 15 sensors. I'll send you the files asap.

Can you confirm that you didn't experienced this type of erroneous rotation issues with the spinal tests?

PS: I'll be at the lighting talks of the Blender conference this friday talking about how the beta testing is going. Keep an eye on it if you can 😉