Juice version: 26.31.0
Juice tends to run with decent performance. When Juice is slow I am typically accessing it via remote proxy and I can relate the issue to my internet speed being slow.
However since two weeks I am noticing that Juice runs very slowly when I have a proxy kernel to the device service open. My firefox detects that the tab I use to access Juice is slowing firefox down:
This tends to clear up after a minute or so.
When the firefox banner disappears, the kernel-status indicator of the device proxy services starts to rapidly blink. This phase takes on the order of 10 minutes. Any code I try to run on the device proxy kernel seems to get stuck in a queue of some sort:
When this is happening it impacts the jupyterlab environment. When the firefox banner is showing, the Juice dashboard and related pages are also affected.
Restarting juice using juice install --restart resolves the problem. But it starts slowly reappearing over time.
Any ideas as to what is causing this?
Hello Tim,
Thank you for your question. I cannot fully reproduce the problem yet, so I want to understand how the slowdown starts.
Did you run specific experiments before the slowdown? Does the page get slow immediately after you use juice install --restart? What is the kernel status on the Task Manager page? Does it show “Connecting …”? Is the kernel still slow after you restart it from the Task Manager?
On our side, we found one possible cause. We changed the restart button in JupyterLab so that it restarts remote kernels.
Best,
Kristian Gogora
Hi Kristian,
First to clear something up. Juice is slow before running juice install --restart, not after.
No specific experiments were run.
Device kernel LED in the task manager shows available.
Kernel is still slow after restarting it via the task manager
Hope this helps!
Thank you for your patience Tim,
The bug has been resolved in the latest release of Juice v26.33.0. If you encounter any issues with Juice feel free to reach out. We will try to help as soon as we can.
1 Like
Hi all,
This problem has not been completely fixed.
This problem still occurs when I send tasks from my user container to the device service, using the `client.execute_blocking` interface.
If I open the proxy kernel to the device service after these tasks have been sent, I can see the blinking status indicator again and I cannot run anything on the proxy kernel.
The time it takes for this to clear up seems to be proportional to the amount of tasks that have been run before I access the proxy kernel.
Hi @kristian, could you take another look at this?
Hi Tim,
Sorry for the late response, we have been investigating this issue last Friday and we did manage to reproduce the blinking issue again, but not yet with client.execute_blocking.
Apparently the issue with the blinking this time is that jupyterlab console is coded to retrieve all of the messages previously sent by any user to the shared service one by one. This is the main cause of the blinking,
I will update you when we find out what went wrong with client.execute.
Best,
Kristian
Hi Kristian, thank you for the reply! Looking forward to hear what you find
I just also remembered that if you send a Task from the same service into itself the service will be indefinetely stuck.
ex:
# kernel: device
from orangeqs.juice import Client
from orangeqs.juice.schemas.tasks import Ping
client = Client()
client.execute_blocking("device", Ping())
This code would make your service stuck.
I am not doing this. I’m sending code from my personal server to the device service.
Hi Tim,
We have found and fixed a bug that was causing the kernel to slow down, however we are still investigating the other issue with the Tasks. However we currently know if the kernel is busy, jupyterlab is unable to estabilish new connections to it. So if a Task makes it busy this syndrome is trigerred.
The fix will be included in the next release of Juice.
Best,
Kristian
1 Like
